lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 44.2392 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.053 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0057 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0092 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/pipeline_orig.qgraph DEEP/20220821/bias/cpBias/20240306T224255Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.9262 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 11.7692 seconds; current memory usage: 0.329 Gibyte, delta: 0.150 Gibyte, peak delta: 0.138 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.329 Gibyte (main), 0.329 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0547 seconds; current memory usage: 0.329 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.329 Gibyte (main), 0.329 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1426 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220821_bias_cpBias_20240306T224255Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2731 seconds; current memory usage: 0.329 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.329 Gibyte (main), 0.329 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1428 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.8706 seconds; current memory usage: 0.332 Gibyte, delta: 0.003 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.332 Gibyte (main), 0.329 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 41f54c57-4a0b-4b8b-b7f5-c52f4f40893e parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-06 22:44:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-06 22:44:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-06 22:44:16 proc_lsst.multi:170 [INFO] [multi] job_id 76135 proc_lsst.multi INFO: [multi] job_id 76135 2024-03-06 22:44:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:16 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 76135 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 76135 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 76135 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 34da67861d542147a42721729950bb24 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b82063eb-abb0-4868-99d0-6c1284d28760_biasIsr_1123730_47.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b82063eb-abb0-4868-99d0-6c1284d28760_biasIsr_1123730_47.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 27778179af1b346358f08b32e588cd5e parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cebe659c-b9c5-49ff-9714-b240e374f798_biasIsr_1123837_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cebe659c-b9c5-49ff-9714-b240e374f798_biasIsr_1123837_47.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 94c61b55af861d34963685a5eb87ce35 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8c7215cf-eb51-4b3f-b1ba-ce6a064ec660_biasIsr_1123732_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8c7215cf-eb51-4b3f-b1ba-ce6a064ec660_biasIsr_1123732_47.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 01fdb637480ff12f6b0e544af7139b9e parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/228df216-055c-4abe-8b4c-530cd2dc9916_biasIsr_1123736_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/228df216-055c-4abe-8b4c-530cd2dc9916_biasIsr_1123736_47.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 92d2c8f57888aa229b386038cd22b867 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b31f9cdc-2be6-42ac-8924-cfa5654e0586_biasIsr_1123835_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b31f9cdc-2be6-42ac-8924-cfa5654e0586_biasIsr_1123835_47.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7cad518efa9945364955cf9de8442cf3 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1da186b6-58cd-4058-86de-f4bc7eab6a76_biasIsr_1123838_47.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1da186b6-58cd-4058-86de-f4bc7eab6a76_biasIsr_1123838_47.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash c4e0dcb56351862d2cff0977b4523017 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1cad6b7f-0fb2-4953-97b8-8c86ed50b6eb_biasIsr_1123729_47.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1cad6b7f-0fb2-4953-97b8-8c86ed50b6eb_biasIsr_1123729_47.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 3b1e0a9fd3ec087d4e4bcdb7d1ca823d parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0f6a620d-a83c-4b68-8744-a9f5b0c01f5c_biasIsr_1123731_47.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0f6a620d-a83c-4b68-8744-a9f5b0c01f5c_biasIsr_1123731_47.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 6cc9344b2ad1b094ce1d3494eac51255 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00de9571-108b-4d8f-a6fe-305288de808c_biasIsr_1123830_47.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00de9571-108b-4d8f-a6fe-305288de808c_biasIsr_1123830_47.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6b6812cbedba98a0a661bcd37052d0f8 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/febb4158-094b-4c79-94ef-9c0662918291_biasIsr_1123728_47.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/febb4158-094b-4c79-94ef-9c0662918291_biasIsr_1123728_47.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash faa762e88f43d4b47c73f7fb65f98366 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/00ff130c-13a9-46ba-9413-bdd64bb7b20d_biasIsr_1123734_47.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/00ff130c-13a9-46ba-9413-bdd64bb7b20d_biasIsr_1123734_47.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a4edde5e12105d20e0feabcdd4325dc9 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cc27f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/d5addc9c-3afd-48d9-b7e9-5bc6264f9151_biasIsr_1123738_47.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/d5addc9c-3afd-48d9-b7e9-5bc6264f9151_biasIsr_1123738_47.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash c5717c69642806b5220ded9282c773c6 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5782c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/56c80a84-185d-4894-bec6-db8b129a16f1_biasIsr_1123836_47.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/56c80a84-185d-4894-bec6-db8b129a16f1_biasIsr_1123836_47.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash b82f77e8f9fdcf52354b8d642ba0e032 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5785e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/86ab7a95-ed63-4261-9b88-80e0c57d8ce4_biasIsr_1123833_47.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/86ab7a95-ed63-4261-9b88-80e0c57d8ce4_biasIsr_1123833_47.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 1abb8e1d2a9fcde4901bdd394a7729e1 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5787c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e1067e08-95eb-419f-9646-5965b7287703_biasIsr_1123831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e1067e08-95eb-419f-9646-5965b7287703_biasIsr_1123831_47.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash a28e356597808919c0927a7a30aba0a7 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5784a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/86e78c7d-da31-413a-b18a-b8365217d15f_biasIsr_1123737_47.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/86e78c7d-da31-413a-b18a-b8365217d15f_biasIsr_1123737_47.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e7f625f010b8a5de97d56ca55db47d58 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/bd944be8-8265-4fb3-9059-2240585f6e0a_biasIsr_1123829_47.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/bd944be8-8265-4fb3-9059-2240585f6e0a_biasIsr_1123829_47.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash ec2d9f4441d4ab0fe5c0fa83f207265b parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/081691c3-3973-44c5-9b13-25852248af7b_biasIsr_1123735_47.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/081691c3-3973-44c5-9b13-25852248af7b_biasIsr_1123735_47.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 74f3b50c581b7b8387dc30124d8b9882 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5789a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a7649d44-ba96-403d-9980-058edf13514b_biasIsr_1123832_47.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a7649d44-ba96-403d-9980-058edf13514b_biasIsr_1123832_47.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash b348df112bfd52e75e89d5b066071224 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d843bbe8-07f0-414f-a15a-ad6a236ced25_biasIsr_1123839_47.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d843bbe8-07f0-414f-a15a-ad6a236ced25_biasIsr_1123839_47.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 93b4dea0dceafc60284066bda903b0ee parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/93658629-865d-4ad1-91fc-e4d48a54a3cc_biasIsr_1123834_47.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/93658629-865d-4ad1-91fc-e4d48a54a3cc_biasIsr_1123834_47.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 95678c05bdcded74d4525faaf430787f parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7f9f12cb-a130-4f07-8df8-4c2ac2f3512d_biasIsr_1123733_47.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7f9f12cb-a130-4f07-8df8-4c2ac2f3512d_biasIsr_1123733_47.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App cpBiasCombine, waiting on task 0, task 1, task 2, task 3, task 4, task 5, task 6, task 7, task 8, task 9, task 10, task 11, task 12, task 13, task 14, task 15, task 16, task 17, task 18, task 19, task 20, task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash fdc7d12da4fab61b177e7079fe681486 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4aec241c-1089-4307-85f8-a8e6f025418d_biasIsr_1123835_8.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4aec241c-1089-4307-85f8-a8e6f025418d_biasIsr_1123835_8.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 736eb51af4dd95a084096c48a9ebe39c parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5799e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/371b633c-aee7-447c-a164-fe8f0d13a737_biasIsr_1123832_8.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/371b633c-aee7-447c-a164-fe8f0d13a737_biasIsr_1123832_8.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash f4d3c246896395a93e58dcca61664647 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ce93ad6b-6d67-4c34-b9ce-4e6d20d540b5_biasIsr_1123731_8.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ce93ad6b-6d67-4c34-b9ce-4e6d20d540b5_biasIsr_1123731_8.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 77dab41b85833a733f8b7f6e7f89fac7 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/bf504e3e-b733-4eba-9716-31f136e9dff4_biasIsr_1123735_8.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/bf504e3e-b733-4eba-9716-31f136e9dff4_biasIsr_1123735_8.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 5d205e750a34f2564bfa595843cc7f9b parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fffe3773-5214-456e-882b-327980d225bc_biasIsr_1123837_8.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fffe3773-5214-456e-882b-327980d225bc_biasIsr_1123837_8.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9f2ed931af3729e13b224231fdf3c5bd parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/54522816-2b74-4af4-b7fa-7b5a33cde0c5_biasIsr_1123737_8.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/54522816-2b74-4af4-b7fa-7b5a33cde0c5_biasIsr_1123737_8.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash affe6b447797f020279dd36c4b220d7a parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a7bfc1e9-00d6-4a68-afc0-147a04eef777_biasIsr_1123836_8.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a7bfc1e9-00d6-4a68-afc0-147a04eef777_biasIsr_1123836_8.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 19e07a2671a2fc393e8bdaa12010a573 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb312fec-cc1a-4ac1-9ad4-50a93b155d8e_biasIsr_1123839_8.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb312fec-cc1a-4ac1-9ad4-50a93b155d8e_biasIsr_1123839_8.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash f57a57b0c214b700cf74c57c90131035 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aa27dfca-e90d-4579-b6a8-a5af37caf47b_biasIsr_1123738_8.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aa27dfca-e90d-4579-b6a8-a5af37caf47b_biasIsr_1123738_8.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 2ae08f173c446c6ace811ae7fa88e1c9 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb579f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1485d868-1a9c-4c0b-9e8c-08790b7c8920_biasIsr_1123833_8.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1485d868-1a9c-4c0b-9e8c-08790b7c8920_biasIsr_1123833_8.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 8cd34826d8604e24b4047cae8b404400 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/158d77b0-5760-4708-ab77-adec3d41d89d_biasIsr_1123733_8.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/158d77b0-5760-4708-ab77-adec3d41d89d_biasIsr_1123733_8.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 464a683a61e426528b8d8e1dcfc4b018 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a27308c-a666-4e61-b670-b3e603d61632_biasIsr_1123736_8.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a27308c-a666-4e61-b670-b3e603d61632_biasIsr_1123736_8.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 734da8249b3695b0cbf3d74de0ee4840 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cb17353f-782b-4df0-beaa-6bf60f5acc4d_biasIsr_1123830_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cb17353f-782b-4df0-beaa-6bf60f5acc4d_biasIsr_1123830_8.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 6dd6133455d1e55d836e7aedf308ad47 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5af87610-064a-46b0-96d3-f24e1092f8cb_biasIsr_1123728_8.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5af87610-064a-46b0-96d3-f24e1092f8cb_biasIsr_1123728_8.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 33af4f607cae79c4d316a6c166a33f55 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5d2ff061-ab72-40ca-ad82-da603f900c61_biasIsr_1123829_8.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5d2ff061-ab72-40ca-ad82-da603f900c61_biasIsr_1123829_8.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash bdb2cc7e13ca13bcf198381a8651f420 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6d018533-2aaf-4590-8db4-dfead4b62eae_biasIsr_1123734_8.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6d018533-2aaf-4590-8db4-dfead4b62eae_biasIsr_1123734_8.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 86d5c25f2bc3b5633ade5c252cb8155c parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/586195a1-d5d4-4c5b-83a2-4e455b4d74aa_biasIsr_1123834_8.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/586195a1-d5d4-4c5b-83a2-4e455b4d74aa_biasIsr_1123834_8.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash eb8d6e3af61066738756939668dff344 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7448ac97-e29e-4a1d-b181-7682711d514e_biasIsr_1123729_8.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7448ac97-e29e-4a1d-b181-7682711d514e_biasIsr_1123729_8.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 863de54bea2438740aa5f7e56e6c6820 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/117a2f12-27d4-4b78-a7e3-c40db6603bef_biasIsr_1123838_8.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/117a2f12-27d4-4b78-a7e3-c40db6603bef_biasIsr_1123838_8.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 0b08a3f39b1c8673a9a2be114f5534e4 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/665673d3-4e84-4662-990f-58d620ad9681_biasIsr_1123730_8.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/665673d3-4e84-4662-990f-58d620ad9681_biasIsr_1123730_8.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 11f83af8b8c9725504688aca8a155cd8 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/5877146f-ac4a-4939-93bf-dab7b797fb89_biasIsr_1123732_8.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/5877146f-ac4a-4939-93bf-dab7b797fb89_biasIsr_1123732_8.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash dae2afdaa968b2f7f7f33a043d526218 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c3c1210c-4732-4a32-a0a7-8e028ca1db34_biasIsr_1123831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c3c1210c-4732-4a32-a0a7-8e028ca1db34_biasIsr_1123831_8.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App cpBiasCombine, waiting on task 23, task 24, task 25, task 26, task 27, task 28, task 29, task 30, task 31, task 32, task 33, task 34, task 35, task 36, task 37, task 38, task 39, task 40, task 41, task 42, task 43, task 44 parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 5f01c0b99a7d68fbf4031f00d96ea660 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/920f2574-6c87-4ab1-8efe-80ec2b7d5098_biasIsr_1123836_29.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/920f2574-6c87-4ab1-8efe-80ec2b7d5098_biasIsr_1123836_29.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash f803812c1a8f4f895a4e0cf2ea966a1a parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/78a55d6e-8498-4c83-a0c5-8e3ee2637501_biasIsr_1123831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/78a55d6e-8498-4c83-a0c5-8e3ee2637501_biasIsr_1123831_29.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 2b7525121f46507d7a929d75d60bd9e0 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/77eeb1f3-d1a9-41e4-968f-aa2ded0bbf71_biasIsr_1123730_29.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/77eeb1f3-d1a9-41e4-968f-aa2ded0bbf71_biasIsr_1123730_29.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash f29a6defa261e5c56f13d4e0c29cee5b parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a96838ce-1fcb-4095-8d73-94c81a07b4fd_biasIsr_1123736_29.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a96838ce-1fcb-4095-8d73-94c81a07b4fd_biasIsr_1123736_29.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash db98c3e73545de53fe042a6c95783156 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aca1db13-40cf-4c03-9b74-5dad9bc007c5_biasIsr_1123738_29.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aca1db13-40cf-4c03-9b74-5dad9bc007c5_biasIsr_1123738_29.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash e1cc9b9cb9a109a198a5567dd095e656 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fa0b2def-6d07-4f00-8e84-3e1069eedb55_biasIsr_1123732_29.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fa0b2def-6d07-4f00-8e84-3e1069eedb55_biasIsr_1123732_29.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash e731b87a1a31e6014e12cf42bce57ec0 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d41490ac-6c04-4748-afea-6de3952fa049_biasIsr_1123839_29.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d41490ac-6c04-4748-afea-6de3952fa049_biasIsr_1123839_29.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 849c429753de6e34ff781538c7a36308 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/436c068f-6ea9-4506-9b98-5fff088b90c8_biasIsr_1123837_29.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/436c068f-6ea9-4506-9b98-5fff088b90c8_biasIsr_1123837_29.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 40f817c130ea9e1ebc461260408fefd2 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/eef2043b-7622-4f5c-95d5-710b9c55ba08_biasIsr_1123835_29.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/eef2043b-7622-4f5c-95d5-710b9c55ba08_biasIsr_1123835_29.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash f60a1ad6e38b52384856f85ffb3993bd parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ac702e22-7767-42fb-b1c4-0d6f08f8bb55_biasIsr_1123729_29.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ac702e22-7767-42fb-b1c4-0d6f08f8bb55_biasIsr_1123729_29.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash e6dceb3b09e24424dd5c0e849b3d597b parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/b27b065d-390f-4ba8-96f6-56813cdb7e99_biasIsr_1123735_29.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/b27b065d-390f-4ba8-96f6-56813cdb7e99_biasIsr_1123735_29.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 6833cc888b3c3b915113dc013fef7417 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/142bdcf1-070e-4ca0-8397-055f11082b67_biasIsr_1123733_29.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/142bdcf1-070e-4ca0-8397-055f11082b67_biasIsr_1123733_29.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash b7daeeb29634cb5e0a7ec145124ed154 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1c295a06-14d3-4285-951f-37666aab968b_biasIsr_1123728_29.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1c295a06-14d3-4285-951f-37666aab968b_biasIsr_1123728_29.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash b75ebbb27de8781cf9653917cbfc0cb2 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/73927637-d2b5-487a-a0e5-01958b8976c2_biasIsr_1123834_29.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/73927637-d2b5-487a-a0e5-01958b8976c2_biasIsr_1123834_29.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 132e4c94c9a7cf5343babe1a428c0e50 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/8f1186ca-9a5e-4f39-bd27-e3d67bad5ffa_biasIsr_1123829_29.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/8f1186ca-9a5e-4f39-bd27-e3d67bad5ffa_biasIsr_1123829_29.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 774dc69c2fda519242133568119462ef parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb57b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/5d6f05fa-65f1-4c4c-9c91-38a87d4113fd_biasIsr_1123833_29.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/5d6f05fa-65f1-4c4c-9c91-38a87d4113fd_biasIsr_1123833_29.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 91ac0d0f345e035f16a3f00b0e170d90 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/c1570c63-e91d-4a49-8a80-0040b84efaf5_biasIsr_1123838_29.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/c1570c63-e91d-4a49-8a80-0040b84efaf5_biasIsr_1123838_29.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash cc28eac251fe839099c350e088fe119d parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6b22d921-080c-4516-bcd7-d100a836f8f4_biasIsr_1123832_29.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6b22d921-080c-4516-bcd7-d100a836f8f4_biasIsr_1123832_29.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash e4a15b473b61c6d1d550d9fc32ab16f4 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d98a331e-dd8d-43d7-b288-3608fb3b87b0_biasIsr_1123734_29.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d98a331e-dd8d-43d7-b288-3608fb3b87b0_biasIsr_1123734_29.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 4ae9e6aed5f51281f53d442f819c9a79 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57217d22-05d4-4673-87da-ab3db30b255b_biasIsr_1123731_29.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57217d22-05d4-4673-87da-ab3db30b255b_biasIsr_1123731_29.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 9330da5e947a8e5da1d008ca3d49f276 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b790c458-c999-4cfd-82c2-9514b1ff3b04_biasIsr_1123830_29.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b790c458-c999-4cfd-82c2-9514b1ff3b04_biasIsr_1123830_29.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash d2a3c2ec678e7d59175dd0913b79609f parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b3aacfd2-9f8c-467e-84bc-ba6f5f83b0d3_biasIsr_1123737_29.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b3aacfd2-9f8c-467e-84bc-ba6f5f83b0d3_biasIsr_1123737_29.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App cpBiasCombine, waiting on task 46, task 47, task 48, task 49, task 50, task 51, task 52, task 53, task 54, task 55, task 56, task 57, task 58, task 59, task 60, task 61, task 62, task 63, task 64, task 65, task 66, task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash d58576532c5b9ddfab42c6d89f34670a parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a5f20490-2731-4f16-b23d-5bfff871ad39_biasIsr_1123835_58.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a5f20490-2731-4f16-b23d-5bfff871ad39_biasIsr_1123835_58.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash ccdc9c1240fd26e8f7af8d95da6cc955 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3542f783-d2d7-4f29-93ca-4741c4fa10c0_biasIsr_1123731_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3542f783-d2d7-4f29-93ca-4741c4fa10c0_biasIsr_1123731_58.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 281dd5967b632a4758dcb83427021b63 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f4c13f17-8988-4dc4-8543-d950d3f99980_biasIsr_1123735_58.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f4c13f17-8988-4dc4-8543-d950d3f99980_biasIsr_1123735_58.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 8d171873d9e6cc258daf3127311cb116 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/984a637b-fa79-4499-a082-d4d2482e5c77_biasIsr_1123737_58.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/984a637b-fa79-4499-a082-d4d2482e5c77_biasIsr_1123737_58.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 04dd7d64211211d9b09d03bc700c7ff2 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d2d2eb58-71bd-46df-a9a8-55e798640ca4_biasIsr_1123736_58.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d2d2eb58-71bd-46df-a9a8-55e798640ca4_biasIsr_1123736_58.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 3b5b73e005650705f8a6731cd4e74467 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f65a8c2a-7843-4841-9702-292707677496_biasIsr_1123837_58.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f65a8c2a-7843-4841-9702-292707677496_biasIsr_1123837_58.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 2935e9ccd99c1720b0a2c2bd0131b4bf parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6c0b28b6-8e02-4a8a-a2e9-f7bbb08ba360_biasIsr_1123832_58.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6c0b28b6-8e02-4a8a-a2e9-f7bbb08ba360_biasIsr_1123832_58.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash ef203d255cff151b5ee70559c43e753f parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f4891c7a-87c4-45fa-8641-7b6e82671944_biasIsr_1123836_58.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f4891c7a-87c4-45fa-8641-7b6e82671944_biasIsr_1123836_58.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 7f248b59d24dfa4b43c79f143a772ec9 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/042ea9be-47b3-4a04-b2a7-02a4414640e9_biasIsr_1123829_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/042ea9be-47b3-4a04-b2a7-02a4414640e9_biasIsr_1123829_58.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 6d1a3c2180b488b434da0a00ec0e639d parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/ad882217-bea6-4539-9ec2-d8fe73ff62f4_biasIsr_1123830_58.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/ad882217-bea6-4539-9ec2-d8fe73ff62f4_biasIsr_1123830_58.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 09a6866f9edf5e04c6acd52ff39199aa parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e2f15d74-3acb-4892-968e-c3ee4af25b95_biasIsr_1123728_58.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e2f15d74-3acb-4892-968e-c3ee4af25b95_biasIsr_1123728_58.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c4b7618dbc66a34e0a98388b7f8c587d parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/73c1f9f7-9f4b-4526-9aed-df81a1c9f6dd_biasIsr_1123730_58.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/73c1f9f7-9f4b-4526-9aed-df81a1c9f6dd_biasIsr_1123730_58.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash ccb80bb71af6c8349bc52c235b04b3a9 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ea547cf9-645f-403b-aad4-04a97ddd9078_biasIsr_1123729_58.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ea547cf9-645f-403b-aad4-04a97ddd9078_biasIsr_1123729_58.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 4a867ab13ea6af838a160220adafbd7a parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/845f5e01-4b2f-45fd-8752-ba8d0925f65a_biasIsr_1123834_58.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/845f5e01-4b2f-45fd-8752-ba8d0925f65a_biasIsr_1123834_58.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 4df7790c51af0fea39a65407e8b6acdd parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/b7a96819-6371-49f4-963b-72e5ad7e4225_biasIsr_1123732_58.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/b7a96819-6371-49f4-963b-72e5ad7e4225_biasIsr_1123732_58.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 83eeab7783faa61aef9ec21478a2dbf9 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/ef0df29a-2a25-4642-b86a-9616dca43f54_biasIsr_1123738_58.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/ef0df29a-2a25-4642-b86a-9616dca43f54_biasIsr_1123738_58.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash a5a5b2a82c3ca3aa7d0013efccbb9cf3 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/7effdd4f-d9a3-4b9c-aff1-fee482f803db_biasIsr_1123831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/7effdd4f-d9a3-4b9c-aff1-fee482f803db_biasIsr_1123831_58.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash f802609f96cd847523c786c0ffbed5fb parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d0cb63f6-d400-478d-ae8a-acb4c288b131_biasIsr_1123839_58.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d0cb63f6-d400-478d-ae8a-acb4c288b131_biasIsr_1123839_58.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash dfb5007aa17c9bbff047b51eb502d478 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/62cb581d-5728-4872-8718-171bd36c8080_biasIsr_1123838_58.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/62cb581d-5728-4872-8718-171bd36c8080_biasIsr_1123838_58.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 3a3acdef440cc4ca77185c5b3efcbdb5 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/866b78f4-9193-45ea-b17c-4f95d64f2025_biasIsr_1123733_58.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/866b78f4-9193-45ea-b17c-4f95d64f2025_biasIsr_1123733_58.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash d602824f23b73ef7393e9b8fbee21f5d parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/da5b16e2-cc0b-4832-9f48-a69fb59b124c_biasIsr_1123833_58.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/da5b16e2-cc0b-4832-9f48-a69fb59b124c_biasIsr_1123833_58.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 44a2e7b7ae2b311e30b1837388e9ccbc parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/38f868df-14f3-45b3-b436-fc42c5006ed9_biasIsr_1123734_58.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/38f868df-14f3-45b3-b436-fc42c5006ed9_biasIsr_1123734_58.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App cpBiasCombine, waiting on task 69, task 70, task 71, task 72, task 73, task 74, task 75, task 76, task 77, task 78, task 79, task 80, task 81, task 82, task 83, task 84, task 85, task 86, task 87, task 88, task 89, task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 8261bc0c1029625e62c18293af6ddbca parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/9e0514b9-52dc-4290-abc6-570ef1e371d9_biasIsr_1123733_61.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/9e0514b9-52dc-4290-abc6-570ef1e371d9_biasIsr_1123733_61.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash c438465c9b13099075290c9909fb7207 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e8c37f71-9880-4691-b3a1-5bf54d6b71d4_biasIsr_1123738_61.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e8c37f71-9880-4691-b3a1-5bf54d6b71d4_biasIsr_1123738_61.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 9c8322cbcc6daf7dc3f30fc2c38415da parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1a41b86e-c142-4a2d-9b71-1fd953252fbe_biasIsr_1123829_61.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1a41b86e-c142-4a2d-9b71-1fd953252fbe_biasIsr_1123829_61.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 690e92dacf9a0924d2c7f8191261d111 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b62e82e9-984b-4b8e-9806-ca00742a49a9_biasIsr_1123830_61.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b62e82e9-984b-4b8e-9806-ca00742a49a9_biasIsr_1123830_61.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 4ec34925f8b587915cef20bd2a77d9c6 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c53a962e-2916-4dd8-b8dc-d858a0610dfa_biasIsr_1123834_61.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c53a962e-2916-4dd8-b8dc-d858a0610dfa_biasIsr_1123834_61.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash fd7e6117b3036c958aa4cf31131bb6af parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/2b572956-8bac-4c92-9ee3-c42499f38845_biasIsr_1123731_61.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/2b572956-8bac-4c92-9ee3-c42499f38845_biasIsr_1123731_61.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 4b1f3a207e4d6b71c65547aa856f0348 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5b050ef1-649c-435c-b0fd-0e61e72e1774_biasIsr_1123728_61.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5b050ef1-649c-435c-b0fd-0e61e72e1774_biasIsr_1123728_61.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 12360d6ea5f8b038935e664a1244b0b0 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/01489b59-71da-4439-8acd-ac371e256a4a_biasIsr_1123735_61.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/01489b59-71da-4439-8acd-ac371e256a4a_biasIsr_1123735_61.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash bc1db961fd1f34aa6be63ff6634c571c parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/97de9a20-a79b-4595-b550-1fe2309efb37_biasIsr_1123838_61.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/97de9a20-a79b-4595-b550-1fe2309efb37_biasIsr_1123838_61.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 9186e4ea7f419abeee9a7642dbde51e6 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/224ef468-fabc-4007-88d3-742d169ee5b4_biasIsr_1123836_61.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/224ef468-fabc-4007-88d3-742d169ee5b4_biasIsr_1123836_61.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 1cd5e66f41b34421fe3d15c66d44eaf3 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/dd73b603-892e-4383-bb38-c1dff5415670_biasIsr_1123837_61.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/dd73b603-892e-4383-bb38-c1dff5415670_biasIsr_1123837_61.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash f25e172649c45ea9103f0aba2fdcc6d9 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/136a7c21-161e-449f-a61c-4aa49bd5a331_biasIsr_1123734_61.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/136a7c21-161e-449f-a61c-4aa49bd5a331_biasIsr_1123734_61.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 81ac7cef5e3fb77ab0bf1412ff325676 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d7c402b4-ad31-4624-8ee1-dc54c645f623_biasIsr_1123736_61.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d7c402b4-ad31-4624-8ee1-dc54c645f623_biasIsr_1123736_61.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash be8f5616b5c44a75199fdb88b16787c4 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e183e803-49ef-4a15-9a0a-121ed25faf75_biasIsr_1123833_61.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e183e803-49ef-4a15-9a0a-121ed25faf75_biasIsr_1123833_61.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash f0855cb99055dc5185a563000b23c193 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a31af267-326f-4c21-be3a-3aa3cb616137_biasIsr_1123835_61.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a31af267-326f-4c21-be3a-3aa3cb616137_biasIsr_1123835_61.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 9197ee375562916ee0ba553252dc10db parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/dadbb40a-c589-480b-ae88-2bdfbf3db668_biasIsr_1123732_61.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/dadbb40a-c589-480b-ae88-2bdfbf3db668_biasIsr_1123732_61.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash df35d42e3cd67fe704e2b7ec2d75da20 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/c11be066-06d2-49f4-acb1-7c7abadb5e90_biasIsr_1123737_61.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/c11be066-06d2-49f4-acb1-7c7abadb5e90_biasIsr_1123737_61.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash e148399fca77db1da4e54e7c55c04b8c parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4a832dc6-f2ee-47dd-913e-4fe5f1b2923c_biasIsr_1123730_61.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4a832dc6-f2ee-47dd-913e-4fe5f1b2923c_biasIsr_1123730_61.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 315fb899de4f7b2e63c33f42695d970b parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/97800972-6514-495f-b20b-c29c263f47ea_biasIsr_1123832_61.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/97800972-6514-495f-b20b-c29c263f47ea_biasIsr_1123832_61.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 2d477042546fa8209e240e0031488ed7 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/84c53e97-3920-4396-98ca-0c3c2641f030_biasIsr_1123831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/84c53e97-3920-4396-98ca-0c3c2641f030_biasIsr_1123831_61.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 32e89f5bb6d99944afc414d1f20e48ce parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/1a2c8ba1-c6be-49e4-b697-db0bd7bf1a17_biasIsr_1123839_61.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/1a2c8ba1-c6be-49e4-b697-db0bd7bf1a17_biasIsr_1123839_61.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7914861a1eaa6e6b8c57b936e21183e1 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb5a3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/e1371a2d-256a-4a3f-a722-a62d7cb694c3_biasIsr_1123729_61.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/e1371a2d-256a-4a3f-a722-a62d7cb694c3_biasIsr_1123729_61.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App cpBiasCombine, waiting on task 92, task 93, task 94, task 95, task 96, task 97, task 98, task 99, task 100, task 101, task 102, task 103, task 104, task 105, task 106, task 107, task 108, task 109, task 110, task 111, task 112, task 113 parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 39583de599cdf662c858246329adf4eb parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0193c6e1-9e11-4e4a-8fd6-ae4b53224e82_biasIsr_1123732_31.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0193c6e1-9e11-4e4a-8fd6-ae4b53224e82_biasIsr_1123732_31.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash ac23317099545a6ded5e0839878d4a61 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/46da5aaf-5759-4272-90b0-444390affa38_biasIsr_1123831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/46da5aaf-5759-4272-90b0-444390affa38_biasIsr_1123831_31.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 523529d0ce7b97049c521625f81a4e79 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/f92231e9-ba4d-40aa-a3c1-f392fc8589fc_biasIsr_1123728_31.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/f92231e9-ba4d-40aa-a3c1-f392fc8589fc_biasIsr_1123728_31.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash b0eac5bc449b8288a8ef52ec556fcac7 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/8d864be7-fe46-490f-94c8-24e85e0ce1f3_biasIsr_1123838_31.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/8d864be7-fe46-490f-94c8-24e85e0ce1f3_biasIsr_1123838_31.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash e20a17f3033b622de0055ca212eeeeb9 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2bca6ddd-05de-4403-a845-4ff6b27c8a94_biasIsr_1123839_31.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2bca6ddd-05de-4403-a845-4ff6b27c8a94_biasIsr_1123839_31.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 8451152df4303466e5d1bc1789cc9e71 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6e5b2671-9cbc-426f-8dfc-c129c2cc4120_biasIsr_1123834_31.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6e5b2671-9cbc-426f-8dfc-c129c2cc4120_biasIsr_1123834_31.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash d13e5111703afc94b519ed0d93cb2b25 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/2d5c4b49-8971-4122-a7be-c0bdb7feb3ee_biasIsr_1123736_31.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/2d5c4b49-8971-4122-a7be-c0bdb7feb3ee_biasIsr_1123736_31.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 0fa19848014dc756d863bf8cc8cbd84b parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/865bff4a-3b3b-4566-bb64-0455b01b7da8_biasIsr_1123738_31.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/865bff4a-3b3b-4566-bb64-0455b01b7da8_biasIsr_1123738_31.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash cf5bd5440290ce5a036d0bbedc264344 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fa1ca36e-8087-4bb6-a906-0cd91524f17b_biasIsr_1123832_31.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fa1ca36e-8087-4bb6-a906-0cd91524f17b_biasIsr_1123832_31.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 0d2bc807f351adab8fa023261f832816 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/437de724-4331-4ce9-8225-d5a9b9ddae6b_biasIsr_1123735_31.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/437de724-4331-4ce9-8225-d5a9b9ddae6b_biasIsr_1123735_31.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash f5389f396f0630b3f951460b2e880db4 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/8613221c-e465-4c68-bad5-7ed31b1e63a3_biasIsr_1123835_31.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/8613221c-e465-4c68-bad5-7ed31b1e63a3_biasIsr_1123835_31.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 65698e7f105bd51a9beee47345527f55 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/e1ca91e1-d66e-401b-a892-3ebf87e09159_biasIsr_1123737_31.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/e1ca91e1-d66e-401b-a892-3ebf87e09159_biasIsr_1123737_31.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash bb318d10619c3dbb21588114debba48c parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/3129de62-d75c-471a-ae0d-82c6e7ea2b88_biasIsr_1123734_31.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/3129de62-d75c-471a-ae0d-82c6e7ea2b88_biasIsr_1123734_31.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 3e1ee46f9d782d02aa78f079901e4eac parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/add92abe-e099-40d8-b626-887b28d2b52a_biasIsr_1123836_31.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/add92abe-e099-40d8-b626-887b28d2b52a_biasIsr_1123836_31.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 511e747dcf603cb821dcc2c2b2844df4 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/35814946-6647-483d-850e-327d85d21fdf_biasIsr_1123829_31.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/35814946-6647-483d-850e-327d85d21fdf_biasIsr_1123829_31.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash ecc2bddd630d35c5c2856153f7e893c3 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/efd38d84-556c-4a30-a6e4-f428ca51a315_biasIsr_1123731_31.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/efd38d84-556c-4a30-a6e4-f428ca51a315_biasIsr_1123731_31.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 83e01970df31b6aa212086e9091348ab parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8494016c-b7a9-4541-806b-ac3f8732c295_biasIsr_1123733_31.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8494016c-b7a9-4541-806b-ac3f8732c295_biasIsr_1123733_31.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 3d745663c7dc23700f33333925072e04 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2ea4a57c-b6c1-45f1-9a90-cd2cd3bcc3ac_biasIsr_1123730_31.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2ea4a57c-b6c1-45f1-9a90-cd2cd3bcc3ac_biasIsr_1123730_31.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 0d0798296b40b187db9c9e35b7e9796a parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/b41f3bc1-122a-41b0-ad3b-1d332f44a609_biasIsr_1123837_31.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/b41f3bc1-122a-41b0-ad3b-1d332f44a609_biasIsr_1123837_31.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash a5a4906b68e7bb38dc42cb5fccb11676 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e11b549c-d538-4ae2-85ae-52f05325a49f_biasIsr_1123833_31.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e11b549c-d538-4ae2-85ae-52f05325a49f_biasIsr_1123833_31.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 6240c353033f13acd5a6d6ce8244fa7b parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4873f6f2-3c86-40d5-9156-3ca6a0414e28_biasIsr_1123729_31.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4873f6f2-3c86-40d5-9156-3ca6a0414e28_biasIsr_1123729_31.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash eeb06c5f1336226c5be989e10bf61d38 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/5f7f0046-82c9-47fc-89ea-ed9f77623828_biasIsr_1123830_31.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/5f7f0046-82c9-47fc-89ea-ed9f77623828_biasIsr_1123830_31.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App cpBiasCombine, waiting on task 115, task 116, task 117, task 118, task 119, task 120, task 121, task 122, task 123, task 124, task 125, task 126, task 127, task 128, task 129, task 130, task 131, task 132, task 133, task 134, task 135, task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 28c099283d338e618567dc351a3cdd29 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/02475ae0-e6c8-4b98-b410-aac60d538b92_biasIsr_1123728_46.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/02475ae0-e6c8-4b98-b410-aac60d538b92_biasIsr_1123728_46.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 10adecb7f6a8df9f1d3775b8c1e67796 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5eacee1c-9e90-43cf-aef4-09f76e6a89ec_biasIsr_1123736_46.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5eacee1c-9e90-43cf-aef4-09f76e6a89ec_biasIsr_1123736_46.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 7718229eb2c3ef01f42c5e4103d383de parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ae692243-3db1-4d31-9d86-e3b6da4bc7cf_biasIsr_1123730_46.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ae692243-3db1-4d31-9d86-e3b6da4bc7cf_biasIsr_1123730_46.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash c93a48d019170a6578b3d7442e28713c parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/aa00d573-0c92-43e7-8f2a-21667f00613a_biasIsr_1123733_46.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/aa00d573-0c92-43e7-8f2a-21667f00613a_biasIsr_1123733_46.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 249231e10212363ad493593c95554f85 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a569dd7a-312b-45cb-b0b6-914f4003b607_biasIsr_1123731_46.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a569dd7a-312b-45cb-b0b6-914f4003b607_biasIsr_1123731_46.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 6518927c4d9f70f43f7a4dabea2bcde6 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e83d51ef-9f79-48cf-bdd6-58fdc6c5c9fc_biasIsr_1123738_46.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e83d51ef-9f79-48cf-bdd6-58fdc6c5c9fc_biasIsr_1123738_46.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 5bed18e633ef36de0737707ebcc69648 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/48802482-cb8e-4cc0-8171-7b77deef22f9_biasIsr_1123831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/48802482-cb8e-4cc0-8171-7b77deef22f9_biasIsr_1123831_46.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 6e502b0d0aaeb14b5f831f26a62bf9c4 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c150778b-6ea9-4ade-9d6c-0086b9f6b513_biasIsr_1123729_46.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c150778b-6ea9-4ade-9d6c-0086b9f6b513_biasIsr_1123729_46.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash bbb114fe0c78e175842748f6228390e1 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/75a7f848-98de-4207-a22d-d37ba4a590c7_biasIsr_1123832_46.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/75a7f848-98de-4207-a22d-d37ba4a590c7_biasIsr_1123832_46.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash df993c08a2b25efda04f5d5bf9147b65 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/4c572107-1a9c-4a0a-b1f4-84deb7196a76_biasIsr_1123735_46.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/4c572107-1a9c-4a0a-b1f4-84deb7196a76_biasIsr_1123735_46.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 4ef72c2b6ae08a97975fbefeb3c6835d parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/0a3a9f26-ae43-4d06-9256-25f05497ee7e_biasIsr_1123834_46.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/0a3a9f26-ae43-4d06-9256-25f05497ee7e_biasIsr_1123834_46.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 9682cb799e9e88bb44c2f8d098651f56 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1ff2f489-ea30-43d6-af13-352de76f57ad_biasIsr_1123835_46.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1ff2f489-ea30-43d6-af13-352de76f57ad_biasIsr_1123835_46.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash a2f583e86d59f558589fa479412ec8cc parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/1304124c-6073-40f6-889c-0f4387388604_biasIsr_1123837_46.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/1304124c-6073-40f6-889c-0f4387388604_biasIsr_1123837_46.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 49fc0b00b029faa4594962fb950bf645 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-06 22:44:17 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8c82c370-d52f-44ed-8b3a-6d80b4d410c2_biasIsr_1123734_46.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8c82c370-d52f-44ed-8b3a-6d80b4d410c2_biasIsr_1123734_46.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 75acb48f63fc1fa1ac9c6001bb43a90c parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/4c14a19e-b19f-4f68-b8cc-3916d6f9a816_biasIsr_1123830_46.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/4c14a19e-b19f-4f68-b8cc-3916d6f9a816_biasIsr_1123830_46.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash e44bfb16def923f0b248d747774e8b20 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/848115d4-c09a-4e31-a70a-1003456891ac_biasIsr_1123839_46.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/848115d4-c09a-4e31-a70a-1003456891ac_biasIsr_1123839_46.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash b483e4ad667f2ea14075d777d67dfd7e parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/57a39e40-0ea9-4245-9adc-614a66d42377_biasIsr_1123737_46.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/57a39e40-0ea9-4245-9adc-614a66d42377_biasIsr_1123737_46.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 316b91ceb9d48bf78c506cef987ac3d4 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1c682e37-8e48-48cf-af3d-52f935171dea_biasIsr_1123836_46.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1c682e37-8e48-48cf-af3d-52f935171dea_biasIsr_1123836_46.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash d739565fa4dbe5309d7897eeb329ed28 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fe715f08-850e-4bfe-8c1a-250b1711b91c_biasIsr_1123732_46.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fe715f08-850e-4bfe-8c1a-250b1711b91c_biasIsr_1123732_46.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 18bd3e8b611a2e204f6d7268c50145ed parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/7feb3229-1c05-4434-83e4-71cffe9a2843_biasIsr_1123829_46.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/7feb3229-1c05-4434-83e4-71cffe9a2843_biasIsr_1123829_46.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 6c6856a012abff2b4b1d499e989f7374 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 151 parsl.dataflow.strategy DEBUG: Executor multi has 151 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1.1709793857.6528943.submit parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2.1709793857.6535084.submit parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3.1709793857.6539736.submit parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4.1709793857.6544309.submit parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5.1709793857.654878.submit parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6.1709793857.655325.submit parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-7.1709793857.655753.submit parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-8.1709793857.6562107.submit parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-9.1709793857.656645.submit parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-10.1709793857.6572425.submit parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-11.1709793857.6578224.submit parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-12.1709793857.658273.submit parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-13.1709793857.6587284.submit parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-14.1709793857.659185.submit parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-15.1709793857.6595995.submit parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-16.1709793857.660046.submit parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-17.1709793857.6604774.submit parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-18.1709793857.660954.submit parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-19.1709793857.6614354.submit parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-20.1709793857.6619298.submit parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-21.1709793857.662385.submit parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-22.1709793857.6628168.submit parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-23.1709793857.6632483.submit parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-24.1709793857.663687.submit parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-25.1709793857.6641302.submit parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-26.1709793857.664546.submit parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-27.1709793857.664967.submit parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-28.1709793857.665378.submit parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-29.1709793857.6658187.submit parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-30.1709793857.6662462.submit parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-31.1709793857.666665.submit parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-32.1709793857.6670976.submit parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-33.1709793857.6675246.submit parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-34.1709793857.6693997.submit parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-35.1709793857.6698482.submit parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-36.1709793857.670311.submit parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-06 22:44:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-37.1709793857.6707845.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/9e251f49-eea2-4000-9083-894307be3d26_biasIsr_1123833_46.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/9e251f49-eea2-4000-9083-894307be3d26_biasIsr_1123833_46.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 5985c013c7bfb86a4c0b14ebc9ec23bb parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f9f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f6427b22-49dc-4313-b2b7-3b270d4db69b_biasIsr_1123838_46.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f6427b22-49dc-4313-b2b7-3b270d4db69b_biasIsr_1123838_46.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App cpBiasCombine, waiting on task 138, task 139, task 140, task 141, task 142, task 143, task 144, task 145, task 146, task 147, task 148, task 149, task 150, task 151, task 152, task 153, task 154, task 155, task 156, task 157, task 158, task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 3ff123043128fee51f7fae0b00e9aed7 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/62173ce2-0a7f-46ff-ba1b-a958f0b174ce_biasIsr_1123835_49.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/62173ce2-0a7f-46ff-ba1b-a958f0b174ce_biasIsr_1123835_49.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ca88c6b00aa7c6aca83979475d6c6465 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/07f10ca3-0c1d-42a7-ac2a-1331c2a718e3_biasIsr_1123836_49.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/07f10ca3-0c1d-42a7-ac2a-1331c2a718e3_biasIsr_1123836_49.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash a42959d3675d4ace708c3dc3ddf5a9b1 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f12c2419-542f-474d-9f57-65252c51f4e3_biasIsr_1123738_49.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f12c2419-542f-474d-9f57-65252c51f4e3_biasIsr_1123738_49.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash a5f5085caf04fc172edd6b9ae145fbae parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/538cf3bb-eece-4026-a519-b43762e80131_biasIsr_1123838_49.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/538cf3bb-eece-4026-a519-b43762e80131_biasIsr_1123838_49.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash a60b229beafdf919727cc1a9e0517714 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5945d231-5735-451f-ab2b-585ceb58796f_biasIsr_1123731_49.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5945d231-5735-451f-ab2b-585ceb58796f_biasIsr_1123731_49.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 2a8bf26cafb2e33cca5dc625a325b72b parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f81e9250-26d8-4680-bd2b-b9c5d98e333e_biasIsr_1123837_49.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f81e9250-26d8-4680-bd2b-b9c5d98e333e_biasIsr_1123837_49.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash c5ea0441787af9fb94a9ac97f75d2f80 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/235c39a5-4190-40d0-89ea-133054c8aba0_biasIsr_1123729_49.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/235c39a5-4190-40d0-89ea-133054c8aba0_biasIsr_1123729_49.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e83a705e2b2740ffac9f030c99572e59 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/41699414-e036-459d-a365-c7d80e01b01d_biasIsr_1123736_49.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/41699414-e036-459d-a365-c7d80e01b01d_biasIsr_1123736_49.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 12bf2f1f150cadaa0f202c500b62b24e parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/052facda-2379-47f4-9fb9-dc2f7eda0bdc_biasIsr_1123830_49.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/052facda-2379-47f4-9fb9-dc2f7eda0bdc_biasIsr_1123830_49.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash e7448eade79b5c0a8c83ad127eaf22fc parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/cc5aa1ab-e6e4-4054-b5b8-bbbc1222356b_biasIsr_1123734_49.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/cc5aa1ab-e6e4-4054-b5b8-bbbc1222356b_biasIsr_1123734_49.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash d9ba981b7d235f43cacb1e8c67fccf85 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fc6fe017-935d-4de5-9215-e238714335be_biasIsr_1123832_49.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fc6fe017-935d-4de5-9215-e238714335be_biasIsr_1123832_49.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 6c3fc0f10bb4737f03ff7b608dddcbc3 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/53b549d4-c70e-4847-889e-8e595e8a5942_biasIsr_1123730_49.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/53b549d4-c70e-4847-889e-8e595e8a5942_biasIsr_1123730_49.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash a220c0c9ca8bf666fea8bc0863e3fb14 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e711bd09-9213-44e8-a211-be2912a2bf5c_biasIsr_1123733_49.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e711bd09-9213-44e8-a211-be2912a2bf5c_biasIsr_1123733_49.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 7edb7fb3981a3a2b11c35e0c86a28790 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/efd9ab96-05a8-4ec3-8859-a43eca10f66b_biasIsr_1123728_49.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/efd9ab96-05a8-4ec3-8859-a43eca10f66b_biasIsr_1123728_49.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 55020425f1ebc932fb4cb288f92f5799 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/7d5962b7-f0b5-4054-9d3e-2a01b9dc5788_biasIsr_1123834_49.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/7d5962b7-f0b5-4054-9d3e-2a01b9dc5788_biasIsr_1123834_49.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash c36328134805d17a2c56e6c0f71ff749 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/015a0e3e-5e27-4eb2-a06d-8a9ae08b7191_biasIsr_1123831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/015a0e3e-5e27-4eb2-a06d-8a9ae08b7191_biasIsr_1123831_49.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 23cdeb42b3c810fed5d01774779e8b32 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/0fe07bbf-68a8-41d9-83b1-b023c185c946_biasIsr_1123735_49.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/0fe07bbf-68a8-41d9-83b1-b023c185c946_biasIsr_1123735_49.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 8567a5c8eb61b3d9d14b0c806a3aa02a parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/60d623c1-f01c-4a28-b998-b8759ae3cc05_biasIsr_1123737_49.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/60d623c1-f01c-4a28-b998-b8759ae3cc05_biasIsr_1123737_49.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash ef3f279d06bca983d26e71d5b802e2d9 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/7f2856d3-d6a7-45d6-910c-22470c5cdfc2_biasIsr_1123833_49.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/7f2856d3-d6a7-45d6-910c-22470c5cdfc2_biasIsr_1123833_49.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash cf256f4e2c303dabc151298754010bb4 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f70275c8-4306-4fc9-becb-7904050cda07_biasIsr_1123732_49.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f70275c8-4306-4fc9-becb-7904050cda07_biasIsr_1123732_49.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 544a1e9b5c6076c613ca60c04d81d220 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/19c10088-c634-4457-81e0-c571c59c0071_biasIsr_1123829_49.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/19c10088-c634-4457-81e0-c571c59c0071_biasIsr_1123829_49.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 743f4c6e0c0ecccd10abe8e1ef689126 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/661c8a77-3aa0-4206-b88a-04c5fa4f201f_biasIsr_1123839_49.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/661c8a77-3aa0-4206-b88a-04c5fa4f201f_biasIsr_1123839_49.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App cpBiasCombine, waiting on task 161, task 162, task 163, task 164, task 165, task 166, task 167, task 168, task 169, task 170, task 171, task 172, task 173, task 174, task 175, task 176, task 177, task 178, task 179, task 180, task 181, task 182 parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 183 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash f3da420a93f638b48b3b1d1120a4ded4 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/56ebf129-3584-4785-a12d-b819186b2fcb_biasIsr_1123829_60.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/56ebf129-3584-4785-a12d-b819186b2fcb_biasIsr_1123829_60.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash f023aa20406f92653b5f1290058e8f0f parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/252f7aed-2a3d-4aa6-ae1c-9c4572333d0b_biasIsr_1123832_60.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/252f7aed-2a3d-4aa6-ae1c-9c4572333d0b_biasIsr_1123832_60.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash e7b03759bd60ed5d69d7f43831524d55 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1691f838-8b99-4db0-89c2-ded9def6dbc1_biasIsr_1123835_60.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1691f838-8b99-4db0-89c2-ded9def6dbc1_biasIsr_1123835_60.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 8554cc08178534eb93e751795ee4c291 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fc9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/eab92520-323e-4629-824b-92eb506bb074_biasIsr_1123833_60.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/eab92520-323e-4629-824b-92eb506bb074_biasIsr_1123833_60.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 598b62673dab838169b2a20ffaefd6fb parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/9e4e1057-b315-43d5-a01a-c93a3aff8c99_biasIsr_1123729_60.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/9e4e1057-b315-43d5-a01a-c93a3aff8c99_biasIsr_1123729_60.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 9357996404f9f863746500f12a762b51 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/23211beb-5f43-44d3-8257-9a1935975e65_biasIsr_1123737_60.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/23211beb-5f43-44d3-8257-9a1935975e65_biasIsr_1123737_60.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash b7aabf30452ff54c02a1f92d3c405928 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d82ff71d-d9da-4a0e-b640-b5b3567e2a1d_biasIsr_1123728_60.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d82ff71d-d9da-4a0e-b640-b5b3567e2a1d_biasIsr_1123728_60.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash dbdcbb94d2594625cae562b908eec1d4 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e5b7159b-54b1-4e73-8d49-b22a73b9f237_biasIsr_1123738_60.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e5b7159b-54b1-4e73-8d49-b22a73b9f237_biasIsr_1123738_60.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 09dd99aa1d5e0cf7080d8edb38f4b354 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/43baf188-1e00-45da-90f9-c10eed90d89b_biasIsr_1123836_60.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/43baf188-1e00-45da-90f9-c10eed90d89b_biasIsr_1123836_60.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 2dd6bf8d3d971e36e57f3b4e2f275796 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/05c0793e-3816-463e-abab-8d637bb4495b_biasIsr_1123838_60.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/05c0793e-3816-463e-abab-8d637bb4495b_biasIsr_1123838_60.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash d51538280a36d8058685af52fb07ba88 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/604279f7-2777-453a-8c8f-3d444b701bbf_biasIsr_1123733_60.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/604279f7-2777-453a-8c8f-3d444b701bbf_biasIsr_1123733_60.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 6237b2472488a6704697698adabf9bb0 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/dde38fc8-8ac9-4c72-ab65-da24a7f9c4b6_biasIsr_1123831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/dde38fc8-8ac9-4c72-ab65-da24a7f9c4b6_biasIsr_1123831_60.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 649349b41a75944f73d50080a9851e6b parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fca980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8d26a1b9-0d75-4286-a336-f93f198645f5_biasIsr_1123734_60.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8d26a1b9-0d75-4286-a336-f93f198645f5_biasIsr_1123734_60.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash bf2b538fc5ce78ef2f85126fa5776201 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/30d7ce11-ae77-4fc7-9080-4b31fe1df28f_biasIsr_1123735_60.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/30d7ce11-ae77-4fc7-9080-4b31fe1df28f_biasIsr_1123735_60.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash d3e44919cfd393ea485e56228abb3dec parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/098e1dff-be63-436a-a8f3-20875f94d6a5_biasIsr_1123839_60.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/098e1dff-be63-436a-a8f3-20875f94d6a5_biasIsr_1123839_60.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash a9e72ad7a070eb44a09077a15c7bef5a parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/dc6b3ba5-848b-456c-968f-b4b036829747_biasIsr_1123736_60.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/dc6b3ba5-848b-456c-968f-b4b036829747_biasIsr_1123736_60.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 5c401b51815eea322d73febf9e594a5d parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/32b1cb03-9b6e-473c-ab3c-c2c73587c390_biasIsr_1123837_60.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/32b1cb03-9b6e-473c-ab3c-c2c73587c390_biasIsr_1123837_60.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 588c590e10eae51c84f08fcda79e9999 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/d6083a85-4726-4e81-9a4b-7ba997f4a90d_biasIsr_1123834_60.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/d6083a85-4726-4e81-9a4b-7ba997f4a90d_biasIsr_1123834_60.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 33b74c3b545377a0049da3cb41a51bae parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f3c16aee-5337-4378-bb71-7aeccbc25b5f_biasIsr_1123830_60.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f3c16aee-5337-4378-bb71-7aeccbc25b5f_biasIsr_1123830_60.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 03052e780da4eeb57414f17b1b69c29a parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c876d616-23f9-4056-bcc3-80d93269e7d1_biasIsr_1123731_60.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c876d616-23f9-4056-bcc3-80d93269e7d1_biasIsr_1123731_60.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 96a9f89eb66a75abdf4b9cb6e7f368a7 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b5774a64-d9f3-4438-9184-ee03b74bee4a_biasIsr_1123730_60.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b5774a64-d9f3-4438-9184-ee03b74bee4a_biasIsr_1123730_60.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 7c4a4929b4774033700fd2fd8f12a174 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/02517249-7efe-424e-8c3e-191e5f0a25f6_biasIsr_1123732_60.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/02517249-7efe-424e-8c3e-191e5f0a25f6_biasIsr_1123732_60.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App cpBiasCombine, waiting on task 184, task 185, task 186, task 187, task 188, task 189, task 190, task 191, task 192, task 193, task 194, task 195, task 196, task 197, task 198, task 199, task 200, task 201, task 202, task 203, task 204, task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 65d80798bd01a1d64965129e85d25b78 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c4691277-ee09-4891-9e5d-22d7eec6247c_biasIsr_1123729_23.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c4691277-ee09-4891-9e5d-22d7eec6247c_biasIsr_1123729_23.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 47bff2691148d8002726854d28006822 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fcb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/9faf32ba-2818-44e6-a40c-b97a2c71f609_biasIsr_1123837_23.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/9faf32ba-2818-44e6-a40c-b97a2c71f609_biasIsr_1123837_23.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash d5759cf9f974ea7edadfe386f2690bae parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8c37ec14-a8d1-48eb-bb82-fbeb144933b1_biasIsr_1123733_23.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8c37ec14-a8d1-48eb-bb82-fbeb144933b1_biasIsr_1123733_23.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 20507009f859d96b2999908f83163569 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/16b92e88-b758-4aa5-b8ff-3719d14e98e5_biasIsr_1123839_23.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/16b92e88-b758-4aa5-b8ff-3719d14e98e5_biasIsr_1123839_23.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash e8bbdb4bdeb21bea38b3b194c150347e parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/72559fd4-91af-4e99-add0-1cef98a35f6c_biasIsr_1123728_23.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/72559fd4-91af-4e99-add0-1cef98a35f6c_biasIsr_1123728_23.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 290f51f2f6e08e5b6ba5df1648184aca parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a7cde19f-d482-4531-8779-7dea1e07d860_biasIsr_1123835_23.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a7cde19f-d482-4531-8779-7dea1e07d860_biasIsr_1123835_23.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 9022def98ea6f6ec1adc67fcb2cd7f3d parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/532c0050-bfee-4965-8a33-1a6afad44ce8_biasIsr_1123830_23.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/532c0050-bfee-4965-8a33-1a6afad44ce8_biasIsr_1123830_23.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 1c2d55fdd5118608b008229ffe0e6cdb parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6568c7f7-eb94-4664-8254-68d433b0279a_biasIsr_1123737_23.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6568c7f7-eb94-4664-8254-68d433b0279a_biasIsr_1123737_23.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 9120cb5fd31680426141743d372c014f parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/37918445-f3fe-48bf-8f69-feb412943354_biasIsr_1123735_23.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/37918445-f3fe-48bf-8f69-feb412943354_biasIsr_1123735_23.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash b755c1ea655b73efa9b7b55b4327a52b parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1bbe24e6-8378-4159-aaad-54574144e0d6_biasIsr_1123829_23.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1bbe24e6-8378-4159-aaad-54574144e0d6_biasIsr_1123829_23.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 04647262f7280821bc910ad9aa470163 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/24000e0a-06f3-4d4f-a62e-b3dc68eeb7bd_biasIsr_1123833_23.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/24000e0a-06f3-4d4f-a62e-b3dc68eeb7bd_biasIsr_1123833_23.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash dcf25878281cde38fbd99361168eedff parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a615bfbd-d62a-4e07-94aa-466d4c9f9457_biasIsr_1123730_23.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a615bfbd-d62a-4e07-94aa-466d4c9f9457_biasIsr_1123730_23.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 1793f6904533bd3c0e279d063ff7168c parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/1d1fd27c-f021-4564-b404-ce7e6b6ef0e7_biasIsr_1123734_23.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/1d1fd27c-f021-4564-b404-ce7e6b6ef0e7_biasIsr_1123734_23.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 39891675a0a7faaf4513760f7ed4ac85 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f54be110-e180-42af-8b65-1df2c5ec3153_biasIsr_1123738_23.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f54be110-e180-42af-8b65-1df2c5ec3153_biasIsr_1123738_23.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 686c908c96d3017631a79320389d3912 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2748990c-1963-4f1c-b2b1-e4f72a97fff5_biasIsr_1123836_23.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2748990c-1963-4f1c-b2b1-e4f72a97fff5_biasIsr_1123836_23.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash ed030950e3c0c2d6f42181d65ea03a50 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ebc5e047-296f-41cd-aac6-b56c25cceba8_biasIsr_1123736_23.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ebc5e047-296f-41cd-aac6-b56c25cceba8_biasIsr_1123736_23.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 908bc77981aebbb961bdbd0968aed973 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/111dcbeb-2048-45d1-b09b-50d9bf62a7ad_biasIsr_1123831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/111dcbeb-2048-45d1-b09b-50d9bf62a7ad_biasIsr_1123831_23.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash f575a829eb83c3601c238a0b10e961ee parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a41a1e94-98b4-4289-a91a-363cf3a8d69b_biasIsr_1123838_23.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a41a1e94-98b4-4289-a91a-363cf3a8d69b_biasIsr_1123838_23.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash d60888dd1e743970ffaad963d75c3306 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/19f198b3-bfa1-40ae-a8c2-7c9fadad2b96_biasIsr_1123834_23.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/19f198b3-bfa1-40ae-a8c2-7c9fadad2b96_biasIsr_1123834_23.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 8b92f4787d9074384ec1c3ed96235f8a parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/019d558a-7e1f-4792-9d96-c3cf5dc83041_biasIsr_1123832_23.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/019d558a-7e1f-4792-9d96-c3cf5dc83041_biasIsr_1123832_23.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 1b3e57dcdba0273beb480115c4d44ed5 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/4511c05a-27fa-4410-bcbc-9994a07b1cf2_biasIsr_1123732_23.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/4511c05a-27fa-4410-bcbc-9994a07b1cf2_biasIsr_1123732_23.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash a3ac60f2f24906760d44953cfd3fe54c parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a503165e-010c-45a8-9ef3-7e1282f528ac_biasIsr_1123731_23.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a503165e-010c-45a8-9ef3-7e1282f528ac_biasIsr_1123731_23.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App cpBiasCombine, waiting on task 207, task 208, task 209, task 210, task 211, task 212, task 213, task 214, task 215, task 216, task 217, task 218, task 219, task 220, task 221, task 222, task 223, task 224, task 225, task 226, task 227, task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash baf50ee58d46b6a9e6e6c73b7b654d4e parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/10b59fcc-543d-4578-9917-8d85eecf77eb_biasIsr_1123837_19.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/10b59fcc-543d-4578-9917-8d85eecf77eb_biasIsr_1123837_19.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 4a6369f380b9ea0b4b6be04ceac919ea parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ae767416-5728-47ac-a806-dc6fb71583d6_biasIsr_1123836_19.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ae767416-5728-47ac-a806-dc6fb71583d6_biasIsr_1123836_19.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash a20cb67b1f56d78efce5b3ddd51fc32f parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/6b1ae1be-b26a-4d7b-be00-9549e405e825_biasIsr_1123831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/6b1ae1be-b26a-4d7b-be00-9549e405e825_biasIsr_1123831_19.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 83beb9d8de8981c743fe40c9a34a2c1a parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/007dd14c-af75-47a0-ac2f-2d9a46df3b52_biasIsr_1123733_19.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/007dd14c-af75-47a0-ac2f-2d9a46df3b52_biasIsr_1123733_19.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 1ef16d6eedadfc7cad98687ab450d3be parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/4d69ce85-07d2-4f3a-b3bb-766cd393d6b6_biasIsr_1123735_19.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/4d69ce85-07d2-4f3a-b3bb-766cd393d6b6_biasIsr_1123735_19.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 2b2d074cf300d38369dc33a06c94216e parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/346bf8ca-ad78-4278-a7b1-ad680526a9b4_biasIsr_1123738_19.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/346bf8ca-ad78-4278-a7b1-ad680526a9b4_biasIsr_1123738_19.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 422fbf0f2ef50ee2ba2ceaa5ca4f649b parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/1cbc9abb-e682-402d-bb33-3b20880d2cc5_biasIsr_1123832_19.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/1cbc9abb-e682-402d-bb33-3b20880d2cc5_biasIsr_1123832_19.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 1e4daefae9047b2f2e03263a82fddae9 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b82f0ef6-990d-4b58-a283-8f5bb39097c3_biasIsr_1123734_19.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b82f0ef6-990d-4b58-a283-8f5bb39097c3_biasIsr_1123734_19.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 1126eaf520f17bd09819ad782c3f166a parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fe9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/fc6c833b-cb8d-47bf-a748-cdedca749caa_biasIsr_1123829_19.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/fc6c833b-cb8d-47bf-a748-cdedca749caa_biasIsr_1123829_19.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 2795752c1053b22ebf6a51531a81eaee parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f34ca19a-455c-47a7-ba3c-82d4dd76a764_biasIsr_1123737_19.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f34ca19a-455c-47a7-ba3c-82d4dd76a764_biasIsr_1123737_19.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 4592fd00a7bf9bb7dd10ca8f5e372b90 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/9a11e05c-2d7d-4334-b932-a832e7664e91_biasIsr_1123730_19.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/9a11e05c-2d7d-4334-b932-a832e7664e91_biasIsr_1123730_19.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 2c02dee22f327eeade95ab375cc3257f parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8ffbda2f-0310-40b9-a439-c3638d522004_biasIsr_1123732_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8ffbda2f-0310-40b9-a439-c3638d522004_biasIsr_1123732_19.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 9d69e965844308a029cbc94ae487f019 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/de8524d0-1c0b-48e4-b82d-3ce2d2a268b9_biasIsr_1123736_19.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/de8524d0-1c0b-48e4-b82d-3ce2d2a268b9_biasIsr_1123736_19.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 6f2c3d8a4bd7334404a2ec7ae53fcdc6 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cdc2525f-e7ce-476c-b55b-aaadf0732cdf_biasIsr_1123833_19.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cdc2525f-e7ce-476c-b55b-aaadf0732cdf_biasIsr_1123833_19.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 06ee5bf281b55065971e10cd912dab84 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/55cf39ab-69ba-44a2-975d-f6a4809eb25d_biasIsr_1123835_19.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/55cf39ab-69ba-44a2-975d-f6a4809eb25d_biasIsr_1123835_19.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash d5441ce58d8da65dae6ff7cb4c81c4a0 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/71860b5f-514f-44ca-8dab-68dd5fc0261c_biasIsr_1123834_19.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/71860b5f-514f-44ca-8dab-68dd5fc0261c_biasIsr_1123834_19.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 0a242531ad5a06678e3570146f01942f parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/9b5f2215-ad10-480c-8a57-72cf6e72dfee_biasIsr_1123839_19.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/9b5f2215-ad10-480c-8a57-72cf6e72dfee_biasIsr_1123839_19.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash e0c89bccabc0ee0224cc130bb2e61e55 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/57ae9b0e-14e0-4f37-9231-8840db6932f4_biasIsr_1123830_19.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/57ae9b0e-14e0-4f37-9231-8840db6932f4_biasIsr_1123830_19.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash b50fe9f644be11dab6aecde3fddf2f5b parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/7e7cf003-c5d5-41ab-bb35-65ecbfa9701c_biasIsr_1123728_19.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/7e7cf003-c5d5-41ab-bb35-65ecbfa9701c_biasIsr_1123728_19.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 85e88df8d89545488ab4be80971d53fd parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/51fa12ce-7678-477d-a62c-a13ecc992ca7_biasIsr_1123731_19.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/51fa12ce-7678-477d-a62c-a13ecc992ca7_biasIsr_1123731_19.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 6080b365bd19e7a29d71b9f8e82f1eb6 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/eb2eb2d6-9f41-4b2a-bd0b-3bc4c461117e_biasIsr_1123729_19.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/eb2eb2d6-9f41-4b2a-bd0b-3bc4c461117e_biasIsr_1123729_19.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash a3934fd4f677932a392cf3d40412a125 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9fea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/84ddc54a-cab1-4a86-ad1c-fe101185a832_biasIsr_1123838_19.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/84ddc54a-cab1-4a86-ad1c-fe101185a832_biasIsr_1123838_19.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App cpBiasCombine, waiting on task 230, task 231, task 232, task 233, task 234, task 235, task 236, task 237, task 238, task 239, task 240, task 241, task 242, task 243, task 244, task 245, task 246, task 247, task 248, task 249, task 250, task 251 parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 252 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 96bd7529a4071898158c82e11284be08 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/114476bc-3bba-4f05-a62e-87b57e6138ae_biasIsr_1123839_37.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/114476bc-3bba-4f05-a62e-87b57e6138ae_biasIsr_1123839_37.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash c5c7603510672f24ec5b9b5e2e58fb42 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/75032979-2a18-407f-bbff-9edd71f3b09d_biasIsr_1123735_37.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/75032979-2a18-407f-bbff-9edd71f3b09d_biasIsr_1123735_37.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 9b9249dce876e018d7f80a65a6f7720b parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/887728a1-eaf5-4e0c-9549-6d7615460c01_biasIsr_1123834_37.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/887728a1-eaf5-4e0c-9549-6d7615460c01_biasIsr_1123834_37.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 0a8ee147e3e57b87f61f9ec7969be049 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/6c7501b1-60ec-48e2-aca1-329e5b721960_biasIsr_1123837_37.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/6c7501b1-60ec-48e2-aca1-329e5b721960_biasIsr_1123837_37.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 6d1cad2254659ad926df31f771bd5f8a parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5bdaaf71-40b6-4a2c-863a-585c55bc7d56_biasIsr_1123836_37.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5bdaaf71-40b6-4a2c-863a-585c55bc7d56_biasIsr_1123836_37.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 0adf6a3e131fa7a1e87cdd83f65528ea parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a8967ee6-08e0-47e6-b9b8-6f8525568449_biasIsr_1123838_37.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a8967ee6-08e0-47e6-b9b8-6f8525568449_biasIsr_1123838_37.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash a51da70a1dece8de3f6b46bb9ad4cdc2 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/3a1f0de7-6891-45d8-b496-3784fa01f923_biasIsr_1123737_37.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/3a1f0de7-6891-45d8-b496-3784fa01f923_biasIsr_1123737_37.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash f1eeb9e8c9c845f0d3c3f7f578461af8 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/d65bfc1e-db8b-47d0-a36c-1397c8e87322_biasIsr_1123830_37.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/d65bfc1e-db8b-47d0-a36c-1397c8e87322_biasIsr_1123830_37.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 77cd08a5e9f2d2790eede26ef362145a parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a048bca9-413b-4727-b3ba-80f3232c57d8_biasIsr_1123731_37.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a048bca9-413b-4727-b3ba-80f3232c57d8_biasIsr_1123731_37.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 4740d4de4fba8298508a80763fc66131 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6862693e-77c9-40af-90ca-32a0b8dd05b6_biasIsr_1123728_37.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6862693e-77c9-40af-90ca-32a0b8dd05b6_biasIsr_1123728_37.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 6f2d86432d84bef589e8dcde78d12de2 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f95568bf-3f68-4629-8364-e9257f7c3817_biasIsr_1123831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f95568bf-3f68-4629-8364-e9257f7c3817_biasIsr_1123831_37.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 72e3bff5f66c2547a1c82afce6f5bf03 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/ad960c5a-3578-47a8-aa02-72c3dd8fa3bc_biasIsr_1123732_37.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/ad960c5a-3578-47a8-aa02-72c3dd8fa3bc_biasIsr_1123732_37.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash b921e2a7e7c3ae7450a6eb7e5560eb43 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9febec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e3870847-c7b7-4547-9f48-a77ca3eb6538_biasIsr_1123734_37.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e3870847-c7b7-4547-9f48-a77ca3eb6538_biasIsr_1123734_37.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash d0d02f5791eb5fb2af2987d290d9c41e parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9feb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d573c86f-864d-4778-9ffd-d41753362f11_biasIsr_1123832_37.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d573c86f-864d-4778-9ffd-d41753362f11_biasIsr_1123832_37.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash bd268afe162c71a39c69cc15db25f9d1 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e310febe-c97b-4c3c-8bab-f849a9946444_biasIsr_1123733_37.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e310febe-c97b-4c3c-8bab-f849a9946444_biasIsr_1123733_37.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 9d3cb22571060549f83c36d9867377c5 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0140e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ca503cf2-9a36-4a7d-92c0-602dffd3bb07_biasIsr_1123736_37.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ca503cf2-9a36-4a7d-92c0-602dffd3bb07_biasIsr_1123736_37.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 2725f0e43b57d6021f3924c57f64312a parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cf978dca-df36-4b94-8d15-297fd86a2b26_biasIsr_1123833_37.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cf978dca-df36-4b94-8d15-297fd86a2b26_biasIsr_1123833_37.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 5ff06fdf17a4715f1486909fef36ac4a parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0142c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/e11c302e-d483-419a-9363-24e3c1c49746_biasIsr_1123729_37.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/e11c302e-d483-419a-9363-24e3c1c49746_biasIsr_1123729_37.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 5690e8f70da9dfb9d406e464cf22acce parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/55c01fbc-3552-4164-84ea-e26c43a2d4da_biasIsr_1123829_37.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/55c01fbc-3552-4164-84ea-e26c43a2d4da_biasIsr_1123829_37.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 95d4980834be55939102d0c1e871622d parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0144a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3e524b55-f4d9-4dbf-90e5-d0d8eb47e880_biasIsr_1123835_37.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3e524b55-f4d9-4dbf-90e5-d0d8eb47e880_biasIsr_1123835_37.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash af330af608d8a5c571846a0a010d89b4 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1d91ec5a-8655-4293-97dc-9c36df6628f6_biasIsr_1123738_37.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1d91ec5a-8655-4293-97dc-9c36df6628f6_biasIsr_1123738_37.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 9e9a8e4de7b18fbed17617c623b0939a parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4240657b-de48-4d9a-a557-1a79653b7d2c_biasIsr_1123730_37.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4240657b-de48-4d9a-a557-1a79653b7d2c_biasIsr_1123730_37.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App cpBiasCombine, waiting on task 253, task 254, task 255, task 256, task 257, task 258, task 259, task 260, task 261, task 262, task 263, task 264, task 265, task 266, task 267, task 268, task 269, task 270, task 271, task 272, task 273, task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 8df9cfa7f861052382819861c5257a63 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/24075997-d8af-4ec8-b629-dbdb19fc7ec0_biasIsr_1123734_1.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/24075997-d8af-4ec8-b629-dbdb19fc7ec0_biasIsr_1123734_1.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 561f8abe38e1f3db1a9b6098f0c67d77 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0153a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1806a8e1-64fe-4415-8982-fcc77fd7e76a_biasIsr_1123833_1.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1806a8e1-64fe-4415-8982-fcc77fd7e76a_biasIsr_1123833_1.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 86d01ad64e6d8f2dc7e40a25200f96b3 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/459d8fae-0b6d-4814-8595-58d70f15d7cd_biasIsr_1123831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/459d8fae-0b6d-4814-8595-58d70f15d7cd_biasIsr_1123831_1.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 8741288c47009162ba3d69fde957b028 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a9850184-2717-4300-a542-245540b6026b_biasIsr_1123733_1.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a9850184-2717-4300-a542-245540b6026b_biasIsr_1123733_1.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash d46fff4c44f48531aa347dbd1c58a2b2 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca014720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/87903b2b-0a55-4044-92de-a58dd0a84cf3_biasIsr_1123832_1.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/87903b2b-0a55-4044-92de-a58dd0a84cf3_biasIsr_1123832_1.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 1f1816458428324cf3faee8f9c1d61b5 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0156c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/483f5c06-d51f-4bb2-9e3f-88ec764742e5_biasIsr_1123728_1.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/483f5c06-d51f-4bb2-9e3f-88ec764742e5_biasIsr_1123728_1.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 403432ced7fa30331439a2f1de68e76b parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/bd1739aa-41cd-4921-8350-90aa5443e3da_biasIsr_1123730_1.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/bd1739aa-41cd-4921-8350-90aa5443e3da_biasIsr_1123730_1.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 358f122b9ebd24a047c10b862b84380a parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b3e6fe59-f809-4f5e-a113-db7ca0f38120_biasIsr_1123834_1.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b3e6fe59-f809-4f5e-a113-db7ca0f38120_biasIsr_1123834_1.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash da4ccd5fad29bb44a9c075bde34ccef6 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5bac5822-7501-401e-ba84-8534a39eb82a_biasIsr_1123829_1.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5bac5822-7501-401e-ba84-8534a39eb82a_biasIsr_1123829_1.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash bee2835813848da8cb4ce40965c2d460 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0158a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6ec5f410-4e02-42df-b557-238751490b51_biasIsr_1123735_1.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6ec5f410-4e02-42df-b557-238751490b51_biasIsr_1123735_1.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 4fe82b9a4c9df3cde500ebe8dce1a6b6 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0159e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a9ffd243-3be0-4f72-863d-f46344eb47d4_biasIsr_1123838_1.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/a9ffd243-3be0-4f72-863d-f46344eb47d4_biasIsr_1123838_1.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 3483ec880945e71d5edb644554de7728 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/db6a02c8-3eed-4764-a3b6-25c45283c288_biasIsr_1123737_1.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/db6a02c8-3eed-4764-a3b6-25c45283c288_biasIsr_1123737_1.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash f509222c015434aa95ccdb8ae9fd322b parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/10e73af4-c3cd-40bd-a3c1-daef7d0e4b82_biasIsr_1123732_1.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/10e73af4-c3cd-40bd-a3c1-daef7d0e4b82_biasIsr_1123732_1.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 61a3d952795052d32e85c80455de91fe parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/04194083-3b0e-4fed-b735-248b06890bc4_biasIsr_1123839_1.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/04194083-3b0e-4fed-b735-248b06890bc4_biasIsr_1123839_1.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 1ca1464c7cee54f26a1b97d70cd4560b parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/d92da5f9-9752-45fb-8980-376b66807a67_biasIsr_1123837_1.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/d92da5f9-9752-45fb-8980-376b66807a67_biasIsr_1123837_1.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash d1f2147ecb36776905b1939cf4245fc4 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/0d594064-fa73-4957-adfb-1f0317f8a706_biasIsr_1123736_1.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/0d594064-fa73-4957-adfb-1f0317f8a706_biasIsr_1123736_1.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 1209ebcfc361069eb7593de72c94e4d9 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e72b9dd9-ffb6-4afd-9356-eda961e6be9e_biasIsr_1123835_1.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e72b9dd9-ffb6-4afd-9356-eda961e6be9e_biasIsr_1123835_1.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 644bdc50bc28aa735a62ab8ae786f3ed parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/3985ef10-60ee-4442-aed9-aea666d35e9d_biasIsr_1123738_1.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/3985ef10-60ee-4442-aed9-aea666d35e9d_biasIsr_1123738_1.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 5977045326c1763fb2fdff85b88e1d4f parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/9fdea23a-f20b-4262-b849-9780f9d3e5c8_biasIsr_1123729_1.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/9fdea23a-f20b-4262-b849-9780f9d3e5c8_biasIsr_1123729_1.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 851153965540654b457b9d960949d7ea parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca015ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b2aeffc5-0846-416a-b77e-79c86b3a1db8_biasIsr_1123836_1.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b2aeffc5-0846-416a-b77e-79c86b3a1db8_biasIsr_1123836_1.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 5b872fb137164d1a5ba6ec608fedc2b0 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca016020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c9c1f053-ea8a-47e3-b908-c317450a7419_biasIsr_1123731_1.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c9c1f053-ea8a-47e3-b908-c317450a7419_biasIsr_1123731_1.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 6c1e3a79e0009ffdcf6e634707a9c97b parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0160c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/22d20081-48d0-4b36-a7f9-e4c31bccdec9_biasIsr_1123830_1.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/22d20081-48d0-4b36-a7f9-e4c31bccdec9_biasIsr_1123830_1.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App cpBiasCombine, waiting on task 276, task 277, task 278, task 279, task 280, task 281, task 282, task 283, task 284, task 285, task 286, task 287, task 288, task 289, task 290, task 291, task 292, task 293, task 294, task 295, task 296, task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 49fa4ad65aaaf0a6e6324c21c55af3da parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/64971bd0-934c-47c8-a3c4-370cf2abed39_biasIsr_1123835_53.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/64971bd0-934c-47c8-a3c4-370cf2abed39_biasIsr_1123835_53.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash c0f720c1d463554fa7379103ffa657e3 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca016fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/2b2dba46-d784-431e-8e1d-fdffb97b5060_biasIsr_1123830_53.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/2b2dba46-d784-431e-8e1d-fdffb97b5060_biasIsr_1123830_53.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 18f5fb6dd37adf68354fb1b716fa7eab parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0171a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/48f7ac2a-d866-4758-a689-9add20211d10_biasIsr_1123832_53.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/48f7ac2a-d866-4758-a689-9add20211d10_biasIsr_1123832_53.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 74947e8bdc2c892b2cb1b2a8ce8bad91 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca016340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/0f1432b6-6679-4efb-b832-619a00a459ec_biasIsr_1123736_53.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/0f1432b6-6679-4efb-b832-619a00a459ec_biasIsr_1123736_53.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 556fddbc7c96bfeaeef65d424314bd59 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/32caff23-139e-41bd-9bfd-0467c9a3edf3_biasIsr_1123733_53.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/32caff23-139e-41bd-9bfd-0467c9a3edf3_biasIsr_1123733_53.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash c597a28d91264cc099da9e1502ac9631 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0172e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/9d6474d6-bba4-45aa-a32a-f64d831d6469_biasIsr_1123731_53.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/9d6474d6-bba4-45aa-a32a-f64d831d6469_biasIsr_1123731_53.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash c62cc46f863705aa14b759b4b13d4334 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/bcb6f13c-7d01-4170-a7b2-02de07d7996a_biasIsr_1123732_53.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/bcb6f13c-7d01-4170-a7b2-02de07d7996a_biasIsr_1123732_53.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 7293afbf121e9e9257de33220b5fa1c2 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/4864a014-4351-4305-bff0-35e1eff6bcf4_biasIsr_1123737_53.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/4864a014-4351-4305-bff0-35e1eff6bcf4_biasIsr_1123737_53.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash c1f40ace1d77cbfe4d37ae9b48608d0b parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0174c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/0532bb7e-406e-4c98-831a-3af7b594e294_biasIsr_1123728_53.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/0532bb7e-406e-4c98-831a-3af7b594e294_biasIsr_1123728_53.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash bd9c7e68db776cca6977f38d22cfcbd1 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e1dc047b-cdd1-4e1a-a793-b676db4915b5_biasIsr_1123829_53.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e1dc047b-cdd1-4e1a-a793-b676db4915b5_biasIsr_1123829_53.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 7cfc4a3293740d57af736d2c7a04efca parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/037b5d83-761d-4168-8d66-e2879d6a3924_biasIsr_1123838_53.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/037b5d83-761d-4168-8d66-e2879d6a3924_biasIsr_1123838_53.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash a14b34a594ecbf8e12634ec6ec52687c parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/0deb1325-d014-4709-98aa-b55fef6d8921_biasIsr_1123834_53.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/0deb1325-d014-4709-98aa-b55fef6d8921_biasIsr_1123834_53.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 967a97faf5864883ff5ebb941ad35ba4 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0176a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a4844b63-5564-42d6-b07c-52ce2ed2eec2_biasIsr_1123836_53.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a4844b63-5564-42d6-b07c-52ce2ed2eec2_biasIsr_1123836_53.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 4884b5fd6e35bcf2b1487b5354daaebf parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0177e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/31acca3f-dbec-4bc4-b209-9e23b1665b31_biasIsr_1123729_53.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/31acca3f-dbec-4bc4-b209-9e23b1665b31_biasIsr_1123729_53.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 916669c933a2f2e86d8b50faa7bebb47 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/51d99c7e-cfeb-48db-9d7d-caaa995539c1_biasIsr_1123833_53.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/51d99c7e-cfeb-48db-9d7d-caaa995539c1_biasIsr_1123833_53.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash bee2115e215c22251bccb4081f12ebcc parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d5e0c47a-a382-48c5-8dc1-b25e909e2d7b_biasIsr_1123735_53.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d5e0c47a-a382-48c5-8dc1-b25e909e2d7b_biasIsr_1123735_53.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 228a6a88d934cd7e17fbd88673cdece5 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ad24c26b-c3d0-4fa8-9d24-17b10f5e144a_biasIsr_1123734_53.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ad24c26b-c3d0-4fa8-9d24-17b10f5e144a_biasIsr_1123734_53.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 313049d18d48697606f10093c6bc7094 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0179c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/49ab4104-4436-45b9-8ad5-16d213ea272e_biasIsr_1123730_53.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/49ab4104-4436-45b9-8ad5-16d213ea272e_biasIsr_1123730_53.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash a4cedcbca64f4b7e0cf353003f54d666 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/99836bf4-51d7-45bf-9166-e43c9f7e6355_biasIsr_1123831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/99836bf4-51d7-45bf-9166-e43c9f7e6355_biasIsr_1123831_53.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash a05f787a9c7b86062643b331a575c776 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/b67afdd6-1f71-4fe6-acb1-f23b46f60388_biasIsr_1123738_53.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/b67afdd6-1f71-4fe6-acb1-f23b46f60388_biasIsr_1123738_53.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 3fe86fd5423454119e99578965a7cb4b parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ebb525ec-4529-468e-bc4a-e9454e0aa136_biasIsr_1123837_53.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ebb525ec-4529-468e-bc4a-e9454e0aa136_biasIsr_1123837_53.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 47db19eeedf58e370e6d72dd6072e9c9 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca017d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b1ab9147-cf54-4c6d-9b02-644c33343a78_biasIsr_1123839_53.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b1ab9147-cf54-4c6d-9b02-644c33343a78_biasIsr_1123839_53.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App cpBiasCombine, waiting on task 299, task 300, task 301, task 302, task 303, task 304, task 305, task 306, task 307, task 308, task 309, task 310, task 311, task 312, task 313, task 314, task 315, task 316, task 317, task 318, task 319, task 320 parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 321 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash feb6bfe832c9a5f632072fa00ad22daa parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b902bdd0-c3f4-4df3-ac6f-21d6c7276c16_biasIsr_1123729_36.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b902bdd0-c3f4-4df3-ac6f-21d6c7276c16_biasIsr_1123729_36.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 7cb426840b5b0eef7e82e0d01eb80f9c parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b8ea38bf-53bc-42bb-b237-3474d9881ecd_biasIsr_1123835_36.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b8ea38bf-53bc-42bb-b237-3474d9881ecd_biasIsr_1123835_36.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash d0f880caa88f40684a32b8e1b860051a parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6491aa8a-bb17-4cfa-a0f8-740eeed66d15_biasIsr_1123838_36.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6491aa8a-bb17-4cfa-a0f8-740eeed66d15_biasIsr_1123838_36.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 5cac8b5ff1b4ad4e8e64e58b72a78e99 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/2987401b-2c99-4150-88df-27a18881b8e1_biasIsr_1123736_36.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/2987401b-2c99-4150-88df-27a18881b8e1_biasIsr_1123736_36.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 9aacaf3ac903985bfc9217e6f36aa550 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fc9e2f50-8736-4b10-8478-dfdb0c03b414_biasIsr_1123837_36.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fc9e2f50-8736-4b10-8478-dfdb0c03b414_biasIsr_1123837_36.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 9e841706925e0f799c9eda3667d56982 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca038fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/36a4233f-0035-4355-bc42-7ccc76d6d0f1_biasIsr_1123730_36.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/36a4233f-0035-4355-bc42-7ccc76d6d0f1_biasIsr_1123730_36.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 2ed6825ba154050e920a2106b742ebbe parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c6ef40ef-88be-445c-8d48-ad7f05f4a68a_biasIsr_1123831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c6ef40ef-88be-445c-8d48-ad7f05f4a68a_biasIsr_1123831_36.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash bc88b3c524d208ae7fe0a17980b2021c parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cfa72787-c315-484e-9128-9bb327f22cd7_biasIsr_1123830_36.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cfa72787-c315-484e-9128-9bb327f22cd7_biasIsr_1123830_36.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash dd8d16f7f4f56e2333b179e6527273c3 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/8c5058c1-be8d-495a-870b-46dad95f8703_biasIsr_1123832_36.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/8c5058c1-be8d-495a-870b-46dad95f8703_biasIsr_1123832_36.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash be9ef4c6ffa3a31f2f49abc68b0494d5 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/cf4601fc-8d27-4ccc-9156-5dfccaebd2b4_biasIsr_1123732_36.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/cf4601fc-8d27-4ccc-9156-5dfccaebd2b4_biasIsr_1123732_36.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 66c7d9fd9e0f919a33410cc7075bc7dd parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4d55073c-6d15-468a-aa77-8d19881b01d1_biasIsr_1123834_36.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4d55073c-6d15-468a-aa77-8d19881b01d1_biasIsr_1123834_36.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 19bd182dc41f2e82c45dd2261fce1391 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/9d3cb4e7-5469-42d8-83df-ec0898fe16f4_biasIsr_1123728_36.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/9d3cb4e7-5469-42d8-83df-ec0898fe16f4_biasIsr_1123728_36.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 29f38f86e34fb376224d854365c20d0e parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fba80bca-0132-4e60-8c6c-31de866c2ad1_biasIsr_1123733_36.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fba80bca-0132-4e60-8c6c-31de866c2ad1_biasIsr_1123733_36.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash bd9c268b8c36dd47a208a4924df9c8c3 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6269ea77-7528-42fe-bd15-02b8b9280a6e_biasIsr_1123829_36.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6269ea77-7528-42fe-bd15-02b8b9280a6e_biasIsr_1123829_36.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 6eb08830b2877cfe688f85c3fba7f50c parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e3651d60-15da-481b-8be4-f65fa8b4c6d5_biasIsr_1123738_36.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e3651d60-15da-481b-8be4-f65fa8b4c6d5_biasIsr_1123738_36.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 4a06c84568d02f14fc13cf9a5fb5bc31 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/f905763a-7626-4e46-9f7e-1307875065cb_biasIsr_1123833_36.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/f905763a-7626-4e46-9f7e-1307875065cb_biasIsr_1123833_36.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 731834bf3758afb4d4599c4a20f86404 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2fdd9683-b498-4fde-b432-c0b88c07e7ef_biasIsr_1123737_36.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2fdd9683-b498-4fde-b432-c0b88c07e7ef_biasIsr_1123737_36.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash ee6d5d152de377678a6a5c1777478dec parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ced2ce81-b45e-4534-be23-42951d9c18e5_biasIsr_1123839_36.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ced2ce81-b45e-4534-be23-42951d9c18e5_biasIsr_1123839_36.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash fc423d2e6fc98075654ae7958ac2fa30 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c1ae70af-3076-4530-9515-954fcbd7fc24_biasIsr_1123735_36.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c1ae70af-3076-4530-9515-954fcbd7fc24_biasIsr_1123735_36.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 2672cbe55fc4da3ca3e135b85163bcc8 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/27bd4a72-7ab8-4181-b14a-8a731ada147c_biasIsr_1123836_36.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/27bd4a72-7ab8-4181-b14a-8a731ada147c_biasIsr_1123836_36.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash e4a5a677e5be5543ea54bc35c8c579df parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/4f73bc5f-521a-448a-a87e-67fa15db255d_biasIsr_1123734_36.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/4f73bc5f-521a-448a-a87e-67fa15db255d_biasIsr_1123734_36.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 27825bdff5ce6c099b13563c3bce15d7 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/119f7a5b-f036-4a6b-b998-fdafd70bdbf1_biasIsr_1123731_36.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/119f7a5b-f036-4a6b-b998-fdafd70bdbf1_biasIsr_1123731_36.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App cpBiasCombine, waiting on task 322, task 323, task 324, task 325, task 326, task 327, task 328, task 329, task 330, task 331, task 332, task 333, task 334, task 335, task 336, task 337, task 338, task 339, task 340, task 341, task 342, task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 380bfd99521279bfe9054ee3ded3bd54 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/2d367fa2-c648-43ba-a1c0-4b06548a5407_biasIsr_1123829_15.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/2d367fa2-c648-43ba-a1c0-4b06548a5407_biasIsr_1123829_15.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 03b4b2d08f9e18beb7ba803032718102 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/bc1c1052-a980-4cd1-bfe8-fcc90d7e07ee_biasIsr_1123838_15.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/bc1c1052-a980-4cd1-bfe8-fcc90d7e07ee_biasIsr_1123838_15.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 542ac132803eb99a498a0773e7ca70c7 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8b3b0065-6975-493e-b49e-9d01d288d792_biasIsr_1123732_15.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8b3b0065-6975-493e-b49e-9d01d288d792_biasIsr_1123732_15.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash c431a1e9980dfb4817a741cae089e849 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca039b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/712b83a1-e0cb-4db9-b483-c93d2cdeda3a_biasIsr_1123831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/712b83a1-e0cb-4db9-b483-c93d2cdeda3a_biasIsr_1123831_15.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash bfd7da03bfb26f2595b2c91abf4605e1 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ed528223-3bcc-459e-bf68-195df4b2d383_biasIsr_1123834_15.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ed528223-3bcc-459e-bf68-195df4b2d383_biasIsr_1123834_15.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash d4a61732bcde86ec0c200978a6223a96 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d59ec2d0-626a-4ae7-9dfb-019245223663_biasIsr_1123835_15.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d59ec2d0-626a-4ae7-9dfb-019245223663_biasIsr_1123835_15.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 0f80a4dd637ea5d02a11f828f9dc2b86 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/22409203-8a40-4d7f-bd30-38ad5a71bf24_biasIsr_1123737_15.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/22409203-8a40-4d7f-bd30-38ad5a71bf24_biasIsr_1123737_15.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash ad1260eea607e08a0bb2d0151e436478 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e0b7c851-228f-4972-82a8-4ab9b1b6f8a3_biasIsr_1123833_15.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e0b7c851-228f-4972-82a8-4ab9b1b6f8a3_biasIsr_1123833_15.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 757cdf68fcf41488b462fe341b25da14 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c80a737a-fdde-453c-9326-3dcffec86cca_biasIsr_1123832_15.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c80a737a-fdde-453c-9326-3dcffec86cca_biasIsr_1123832_15.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash a0d913e6da68cb034867db32dea3777f parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b5175caa-aa76-4bb7-832b-273a0ed3d279_biasIsr_1123830_15.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b5175caa-aa76-4bb7-832b-273a0ed3d279_biasIsr_1123830_15.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 3e2c247d22e60990fd1f22b0eb765c85 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bf829075-a389-47e3-92e9-cec3e02c506a_biasIsr_1123839_15.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bf829075-a389-47e3-92e9-cec3e02c506a_biasIsr_1123839_15.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 181c8b607466d1e8b1e2596882cb3a19 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f129b5a2-5f91-494e-9910-9c63687aa6ca_biasIsr_1123837_15.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f129b5a2-5f91-494e-9910-9c63687aa6ca_biasIsr_1123837_15.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 0c2f400f2bd6c1d063da69ce80b34db2 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4218a464-5289-44cb-b7e2-815514fad04f_biasIsr_1123730_15.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4218a464-5289-44cb-b7e2-815514fad04f_biasIsr_1123730_15.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 69377034f29b4e3c1145c76936422116 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/855bb09b-e8f8-403f-a512-9af5d77a3a6c_biasIsr_1123731_15.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/855bb09b-e8f8-403f-a512-9af5d77a3a6c_biasIsr_1123731_15.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash a8f3fd74c14df6c714d6fc0add4ed5a8 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/5244d5c4-1a09-4305-9c3f-06224887cad1_biasIsr_1123738_15.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/5244d5c4-1a09-4305-9c3f-06224887cad1_biasIsr_1123738_15.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash b81d01c7ff0d67f786b733b6bbb0219f parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8f6117aa-792a-4ec2-a0a1-f732fb8c2f4e_biasIsr_1123734_15.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8f6117aa-792a-4ec2-a0a1-f732fb8c2f4e_biasIsr_1123734_15.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 70611deec77b4bc2a564e044770a97f7 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a657d3c8-7290-488d-b795-e77f786cb994_biasIsr_1123733_15.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a657d3c8-7290-488d-b795-e77f786cb994_biasIsr_1123733_15.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash b8a237d3621a99447e88a23e7d0bc462 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d24ee2db-5b1a-4088-8841-0422a1da781e_biasIsr_1123735_15.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d24ee2db-5b1a-4088-8841-0422a1da781e_biasIsr_1123735_15.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 7469009b84069207c6ee1dcfe3d0689c parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1a15036f-8ad2-4a64-b044-66371f4c1828_biasIsr_1123729_15.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1a15036f-8ad2-4a64-b044-66371f4c1828_biasIsr_1123729_15.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 2a7e1d0cf89c06632131e44fab250946 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/96a0cde2-e892-4597-add7-ea035da0f8e6_biasIsr_1123836_15.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/96a0cde2-e892-4597-add7-ea035da0f8e6_biasIsr_1123836_15.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash efd546214510c4c39965f0d2a686fa39 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5ecef843-ea56-4361-a621-5d88ed3b0c0f_biasIsr_1123736_15.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5ecef843-ea56-4361-a621-5d88ed3b0c0f_biasIsr_1123736_15.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 9f3e1166e987630f6b9cd765d426ac97 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca03b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/ea3b09d0-9a1a-4f3c-b400-15ce7e37ea82_biasIsr_1123728_15.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/ea3b09d0-9a1a-4f3c-b400-15ce7e37ea82_biasIsr_1123728_15.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App cpBiasCombine, waiting on task 345, task 346, task 347, task 348, task 349, task 350, task 351, task 352, task 353, task 354, task 355, task 356, task 357, task 358, task 359, task 360, task 361, task 362, task 363, task 364, task 365, task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash c6afbab0a1a4fb08bb0d5581984e0282 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4da0d6cc-9b1e-4f88-880b-8a90c7acc1de_biasIsr_1123730_14.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4da0d6cc-9b1e-4f88-880b-8a90c7acc1de_biasIsr_1123730_14.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ee4a5b01ea27fff18f1aeeaa467a746b parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f055dbf2-06ce-421f-a036-4959e7b92588_biasIsr_1123838_14.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f055dbf2-06ce-421f-a036-4959e7b92588_biasIsr_1123838_14.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash d11dc91533464449b015b871cc12a84b parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/974f8640-7c02-4036-8032-1ff1d3867448_biasIsr_1123830_14.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/974f8640-7c02-4036-8032-1ff1d3867448_biasIsr_1123830_14.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 6b6b68a4c6fe1c8cb10b35985c770385 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/c30fccfc-5717-4910-8440-c6c00a3c6af0_biasIsr_1123728_14.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/c30fccfc-5717-4910-8440-c6c00a3c6af0_biasIsr_1123728_14.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 1e9a53c45c69e69462f21dafb2aca7ae parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/01b134b8-b5ef-48fc-8b20-fbf9431985ec_biasIsr_1123734_14.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/01b134b8-b5ef-48fc-8b20-fbf9431985ec_biasIsr_1123734_14.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 28016b6fd6812e46b2258290e4a73a05 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3cf39bf7-c5f8-4bd8-b2d2-fec6fd4df838_biasIsr_1123835_14.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3cf39bf7-c5f8-4bd8-b2d2-fec6fd4df838_biasIsr_1123835_14.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 00d1d8d13a788689ee738834da27d8ae parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/321312a7-3836-41e1-971b-a05b229b1af4_biasIsr_1123829_14.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/321312a7-3836-41e1-971b-a05b229b1af4_biasIsr_1123829_14.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash d6e0aef3468c920f99aaaf65245bccd6 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/f3ba8adf-27e2-46a1-ba8c-1a8362ac0060_biasIsr_1123731_14.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/f3ba8adf-27e2-46a1-ba8c-1a8362ac0060_biasIsr_1123731_14.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 1b6c753c7d0e6efa857de84311ce3abf parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/9cf779f1-28f1-4cee-8a06-6d377108930d_biasIsr_1123833_14.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/9cf779f1-28f1-4cee-8a06-6d377108930d_biasIsr_1123833_14.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash d31f83f87ad41b94ac63b6115fb0d9c8 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/889e3673-c3c1-48b1-b57f-4a95f3946c10_biasIsr_1123738_14.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/889e3673-c3c1-48b1-b57f-4a95f3946c10_biasIsr_1123738_14.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 9270695d3aefbe7664cc085744a018e3 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e63c6333-0b5f-4df8-91c2-f7a7c87e1a36_biasIsr_1123832_14.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e63c6333-0b5f-4df8-91c2-f7a7c87e1a36_biasIsr_1123832_14.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash cb70ee0bd51718910554459576881c7f parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/f395eb6a-40f7-4506-854d-aee672408693_biasIsr_1123733_14.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/f395eb6a-40f7-4506-854d-aee672408693_biasIsr_1123733_14.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 29657c91c4037fe55b17c8aa3faba67b parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/35a7fc33-8cd2-4131-93c1-465c95d156f7_biasIsr_1123831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/35a7fc33-8cd2-4131-93c1-465c95d156f7_biasIsr_1123831_14.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 8cf5e4de936e303d40c9123477f82c18 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/7d2af5da-3793-4c08-ad69-4fedbccb927d_biasIsr_1123735_14.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/7d2af5da-3793-4c08-ad69-4fedbccb927d_biasIsr_1123735_14.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 986d78d7f49875b4089ae31cfda7d7b7 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/01140a82-5bb7-4f02-8e0a-88edb37a50f6_biasIsr_1123729_14.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/01140a82-5bb7-4f02-8e0a-88edb37a50f6_biasIsr_1123729_14.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash e73eb3d090193e6f0fe60f1f7073b64b parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5411340c-9024-4f7c-a351-c0bb352bb540_biasIsr_1123736_14.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5411340c-9024-4f7c-a351-c0bb352bb540_biasIsr_1123736_14.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 62ada4a31bc00f19b13fcd6213dc35ad parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/7e27114a-6672-4123-a3f6-c83e854c7ba9_biasIsr_1123836_14.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/7e27114a-6672-4123-a3f6-c83e854c7ba9_biasIsr_1123836_14.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash a3175a8d56ba71e5daf9584b7b63169b parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca064fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/77051ac0-86d0-491b-ad2b-5afd6b25a446_biasIsr_1123732_14.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/77051ac0-86d0-491b-ad2b-5afd6b25a446_biasIsr_1123732_14.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash c0d49c24d25c77f82b21669a65e0882d parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca065080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/ac254b31-4b0a-4eb0-a06c-0e14003b23fe_biasIsr_1123737_14.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/ac254b31-4b0a-4eb0-a06c-0e14003b23fe_biasIsr_1123737_14.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 1ed141452112ab3a5f88b2c9ad3f4756 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/d3ab88fd-97ce-4022-863b-d3495421dc68_biasIsr_1123834_14.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/d3ab88fd-97ce-4022-863b-d3495421dc68_biasIsr_1123834_14.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 2ebaa5730e6e93b1a353a3d305215f2d parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca065120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/f208f235-2c91-4385-bb38-095f45735328_biasIsr_1123839_14.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/f208f235-2c91-4385-bb38-095f45735328_biasIsr_1123839_14.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash abd885a800e6ac87d84b5415c3011561 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca065260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a80dfc72-c770-477b-8b69-1414329acdee_biasIsr_1123837_14.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a80dfc72-c770-477b-8b69-1414329acdee_biasIsr_1123837_14.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App cpBiasCombine, waiting on task 368, task 369, task 370, task 371, task 372, task 373, task 374, task 375, task 376, task 377, task 378, task 379, task 380, task 381, task 382, task 383, task 384, task 385, task 386, task 387, task 388, task 389 parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 390 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 7cf60cbeeab719467ca053175250608e parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/9d6c1ce0-1898-493d-861a-ce7a7b1f017d_biasIsr_1123830_25.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/9d6c1ce0-1898-493d-861a-ce7a7b1f017d_biasIsr_1123830_25.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash ea2c76036798295a426c833d91a210cf parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/fa282234-f74e-4c56-9a4e-84e7f75dc7b7_biasIsr_1123829_25.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/fa282234-f74e-4c56-9a4e-84e7f75dc7b7_biasIsr_1123829_25.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 2e5a5f9bd2c140d38703aeb9957dbe44 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/a9d1468f-72cb-41d7-af01-31e4de84548b_biasIsr_1123734_25.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/a9d1468f-72cb-41d7-af01-31e4de84548b_biasIsr_1123734_25.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 1910dbab1d2680c88c30c534b62d9a58 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a013bdbb-65b2-4a83-9cbe-490bf38cdaf6_biasIsr_1123730_25.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a013bdbb-65b2-4a83-9cbe-490bf38cdaf6_biasIsr_1123730_25.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 5d5ff7dff4dde487154639b4b43fd01d parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca065440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/6b3c72c7-fe7a-41af-85cc-111fd05ac171_biasIsr_1123836_25.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/6b3c72c7-fe7a-41af-85cc-111fd05ac171_biasIsr_1123836_25.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash fca97a83176fa082bdaf48b64922fc3f parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7dc8c04b-51e1-4cec-9fb8-485a64cf247b_biasIsr_1123839_25.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7dc8c04b-51e1-4cec-9fb8-485a64cf247b_biasIsr_1123839_25.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash de9284f3568e22a7f982bc623de67eb2 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/908ed5ea-360c-4518-8314-776b23810611_biasIsr_1123729_25.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/908ed5ea-360c-4518-8314-776b23810611_biasIsr_1123729_25.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 0a8e4daa9d65e41e4d49239e6584f911 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/183afc67-116a-4881-b7de-6e561408d021_biasIsr_1123834_25.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/183afc67-116a-4881-b7de-6e561408d021_biasIsr_1123834_25.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 4e8f1a3bc07719ea709f17b5d01dcd21 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1f2b9d19-5de0-4953-9190-7d6226c2d94c_biasIsr_1123835_25.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1f2b9d19-5de0-4953-9190-7d6226c2d94c_biasIsr_1123835_25.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash e1e58e5e805782dd413de693dcbd983d parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2c78874e-0dd7-477b-b4fc-413ebccaeecc_biasIsr_1123837_25.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2c78874e-0dd7-477b-b4fc-413ebccaeecc_biasIsr_1123837_25.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 9ea7c774afb892be66520afe4d9fd613 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2a860ba2-67e1-42fa-83e6-e99aca7b80fc_biasIsr_1123728_25.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2a860ba2-67e1-42fa-83e6-e99aca7b80fc_biasIsr_1123728_25.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash a5d64931491949b884b49c0f6904e620 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/957b603f-8d73-41ff-9d66-af2f8e8167cd_biasIsr_1123731_25.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/957b603f-8d73-41ff-9d66-af2f8e8167cd_biasIsr_1123731_25.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash aa517f30788cafb1f0e79ac666573cc6 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0262826f-864a-45fc-84fc-e0558c243d94_biasIsr_1123737_25.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0262826f-864a-45fc-84fc-e0558c243d94_biasIsr_1123737_25.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash d0a0b55798e05a7c36e050267ee935cd parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6276a9a9-cd73-4c97-9a42-e96995b7b485_biasIsr_1123832_25.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6276a9a9-cd73-4c97-9a42-e96995b7b485_biasIsr_1123832_25.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 4b3d4350c462f04eaa3adf793e7980c1 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca0668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/0d83d5a6-571a-4260-a6df-560773d84014_biasIsr_1123831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/0d83d5a6-571a-4260-a6df-560773d84014_biasIsr_1123831_25.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 03decd6b1e78aa4f638c55995eba2ac7 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/014bc8b8-6349-4710-9bbd-e815457b0041_biasIsr_1123735_25.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/014bc8b8-6349-4710-9bbd-e815457b0041_biasIsr_1123735_25.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 11eb5ec0e5d4a31ab9bb9f34e44da0da parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/646d3c14-2b29-4ed3-892c-574dd04a75fe_biasIsr_1123732_25.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/646d3c14-2b29-4ed3-892c-574dd04a75fe_biasIsr_1123732_25.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 5d341729b7a9b54b1cd7199df4008968 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e748a4f4-cbb8-423a-be5b-7c69d551b3ec_biasIsr_1123833_25.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e748a4f4-cbb8-423a-be5b-7c69d551b3ec_biasIsr_1123833_25.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 583f00616a25824aa4c4b4d04b20aba2 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/28aba6ef-1576-44f0-bd7a-dfddc770b6cc_biasIsr_1123838_25.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/28aba6ef-1576-44f0-bd7a-dfddc770b6cc_biasIsr_1123838_25.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 746e18d799764059c1a71efdbf81276e parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7d0aedd0-c3ad-4c0f-b273-bc97afc1c218_biasIsr_1123733_25.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7d0aedd0-c3ad-4c0f-b273-bc97afc1c218_biasIsr_1123733_25.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 221e51ff1f7838efbd4fd3826bac1903 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/7af8c67a-dd73-4e49-918e-58a84284093a_biasIsr_1123736_25.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/7af8c67a-dd73-4e49-918e-58a84284093a_biasIsr_1123736_25.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 42fd579248086f92c70f5975f9bfb469 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca066de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/df2bd6c3-4291-4bd2-8e7d-1488ec47d057_biasIsr_1123738_25.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/df2bd6c3-4291-4bd2-8e7d-1488ec47d057_biasIsr_1123738_25.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App cpBiasCombine, waiting on task 391, task 392, task 393, task 394, task 395, task 396, task 397, task 398, task 399, task 400, task 401, task 402, task 403, task 404, task 405, task 406, task 407, task 408, task 409, task 410, task 411, task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 1a9ab31bbaec151e6fb5e4fdcb030183 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca067d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1dd3f52e-7d8d-4301-9883-ba6b91f96aae_biasIsr_1123829_2.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1dd3f52e-7d8d-4301-9883-ba6b91f96aae_biasIsr_1123829_2.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash eade5546f87bf62f279832c359ff65cd parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca067ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9e4ea082-10eb-4a85-9b67-70c66233921c_biasIsr_1123838_2.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9e4ea082-10eb-4a85-9b67-70c66233921c_biasIsr_1123838_2.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 01fa6d9b23c4fe6c7783e8306d57e919 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca067ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/8ec94025-8404-4e1e-828d-9a8678fc0574_biasIsr_1123730_2.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/8ec94025-8404-4e1e-828d-9a8678fc0574_biasIsr_1123730_2.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 2a572df6ca6cb74a6b660f43963edae3 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca067060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a4f90b76-be76-491e-bccf-905c721852ce_biasIsr_1123738_2.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a4f90b76-be76-491e-bccf-905c721852ce_biasIsr_1123738_2.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash d8d724e3986067e8d9307bcd8b20737c parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502ca067e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/da9705d6-8c07-4199-b7f4-ac64e4c0858c_biasIsr_1123835_2.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/da9705d6-8c07-4199-b7f4-ac64e4c0858c_biasIsr_1123835_2.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 97d87c852ad13a4b2d4d51faabfd98fe parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/59ca5175-23df-4a0f-b32f-a57cca6493e8_biasIsr_1123832_2.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/59ca5175-23df-4a0f-b32f-a57cca6493e8_biasIsr_1123832_2.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash f18e28b648ef725cb745885005061036 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/9a50e325-1273-46e6-b49e-cf0f9823c25e_biasIsr_1123735_2.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/9a50e325-1273-46e6-b49e-cf0f9823c25e_biasIsr_1123735_2.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash f92981c28b9cc8d80d599f30a3fe5d5f parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/585467ed-429e-4d31-93ea-1df6fd6721e0_biasIsr_1123830_2.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/585467ed-429e-4d31-93ea-1df6fd6721e0_biasIsr_1123830_2.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 993ed4ca9df7f9c9c2fbf70f96287a31 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/756039f4-a42c-406c-9671-129fab02fcda_biasIsr_1123733_2.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/756039f4-a42c-406c-9671-129fab02fcda_biasIsr_1123733_2.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 161097d133c5f237dc4885fa9c097057 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/be09a46d-719f-4ce9-bcc1-c07da903e3f1_biasIsr_1123836_2.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/be09a46d-719f-4ce9-bcc1-c07da903e3f1_biasIsr_1123836_2.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 057008e1bd7b2881e1273080a65c465d parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/e7d0fe4b-4f6a-446b-afda-5b6467273aea_biasIsr_1123837_2.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/e7d0fe4b-4f6a-446b-afda-5b6467273aea_biasIsr_1123837_2.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash e094598ec8e33599ebfee930e22509dd parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/2e4ed35a-a395-40fe-a96a-717ac84ec8ed_biasIsr_1123831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/2e4ed35a-a395-40fe-a96a-717ac84ec8ed_biasIsr_1123831_2.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash c1be12e5a3a0067fa0d1541151c20c84 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cc2b3f43-2194-4379-9dc2-e20fa41c0ad1_biasIsr_1123737_2.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cc2b3f43-2194-4379-9dc2-e20fa41c0ad1_biasIsr_1123737_2.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash bb015901700fb8069ff85a4d0ae62f4d parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/8f27e78c-24b7-49dd-ba0c-db5ebf1e5a5c_biasIsr_1123839_2.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/8f27e78c-24b7-49dd-ba0c-db5ebf1e5a5c_biasIsr_1123839_2.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 3072834ea1b9bd50f3c5c45d4e008730 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c6bb8262-cbc6-4b57-a722-4d202ae5ff1e_biasIsr_1123729_2.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c6bb8262-cbc6-4b57-a722-4d202ae5ff1e_biasIsr_1123729_2.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 7e582c447fe469f638b608a98cd26319 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1d929d82-f8ad-4855-8c53-5c159f41c067_biasIsr_1123833_2.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1d929d82-f8ad-4855-8c53-5c159f41c067_biasIsr_1123833_2.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 01945294e5cca161eb0f772481cc811d parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6b278c82-ad8c-4a2a-be7f-afdd5e48e6e6_biasIsr_1123728_2.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6b278c82-ad8c-4a2a-be7f-afdd5e48e6e6_biasIsr_1123728_2.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 7d7019233e8d4fbdb442ad826ee074a8 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ad78671c-60ee-4105-b913-32fbd171444d_biasIsr_1123734_2.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ad78671c-60ee-4105-b913-32fbd171444d_biasIsr_1123734_2.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 0b159fcb35a7cadf4d0af2bfa20a1374 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/924dc7e9-465a-4a4c-84a8-3a1ca111e9cf_biasIsr_1123732_2.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/924dc7e9-465a-4a4c-84a8-3a1ca111e9cf_biasIsr_1123732_2.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 540c7918a986f37b5d80c94ee6ce429a parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/4ffcac81-f3b1-48f7-b697-b06f15de990b_biasIsr_1123731_2.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/4ffcac81-f3b1-48f7-b697-b06f15de990b_biasIsr_1123731_2.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash c7c64015270ef97c75b59e88f1674ed8 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bbb732d2-3597-43da-8c6b-a4a4fd123f4e_biasIsr_1123736_2.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bbb732d2-3597-43da-8c6b-a4a4fd123f4e_biasIsr_1123736_2.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash db087a9ae6735e360e43082c07c64c6d parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4c6f2ad8-2ac5-41fb-9337-31b255b8df4c_biasIsr_1123834_2.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4c6f2ad8-2ac5-41fb-9337-31b255b8df4c_biasIsr_1123834_2.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App cpBiasCombine, waiting on task 414, task 415, task 416, task 417, task 418, task 419, task 420, task 421, task 422, task 423, task 424, task 425, task 426, task 427, task 428, task 429, task 430, task 431, task 432, task 433, task 434, task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 570ae062fe9ec4c8ab9e84d72819d7c0 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/8b6d012f-bd3e-43d2-b720-ef7c1c04303f_biasIsr_1123835_39.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/8b6d012f-bd3e-43d2-b720-ef7c1c04303f_biasIsr_1123835_39.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash c9a66b833440336aa53e3e17e849d0c7 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/62a31af4-6f16-4203-a939-54996b0e4b06_biasIsr_1123737_39.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/62a31af4-6f16-4203-a939-54996b0e4b06_biasIsr_1123737_39.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 5025c54aabd59fd0163a6d0425de7c0d parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d67ce669-2195-4b68-a9a1-8d6708503b33_biasIsr_1123839_39.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d67ce669-2195-4b68-a9a1-8d6708503b33_biasIsr_1123839_39.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 276ac453ec35499c172fb4fa990d5b31 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/6f998b36-fb90-456d-aaa2-37b4e45a61aa_biasIsr_1123731_39.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/6f998b36-fb90-456d-aaa2-37b4e45a61aa_biasIsr_1123731_39.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 2bdaf639881389e295036ac973c365ef parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/e982b8b8-fdbc-4da5-810c-16c7834c016e_biasIsr_1123834_39.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/e982b8b8-fdbc-4da5-810c-16c7834c016e_biasIsr_1123834_39.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash b489f6516e53b61683d87cce018107ce parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/35a0d9bc-7315-43db-b005-6a777469b507_biasIsr_1123729_39.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/35a0d9bc-7315-43db-b005-6a777469b507_biasIsr_1123729_39.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash cdf6990b64f03012ed5fdf9f0861ad5c parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/856ed3de-c330-4173-a286-10056af43ad8_biasIsr_1123733_39.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/856ed3de-c330-4173-a286-10056af43ad8_biasIsr_1123733_39.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 8490ad3d44d290d00da5a5e1504db0c7 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ef41c965-32fb-4ef6-917d-6b58bc92ca60_biasIsr_1123838_39.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ef41c965-32fb-4ef6-917d-6b58bc92ca60_biasIsr_1123838_39.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 53b3e57c9599c21f1ce466658d3320a6 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/268cbcc9-e5ee-4fc9-bed9-44b9452eb546_biasIsr_1123830_39.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/268cbcc9-e5ee-4fc9-bed9-44b9452eb546_biasIsr_1123830_39.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash ebcd90d912e68f5411917e8b9c7cbfcd parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/ec19b6d9-8ae6-48b0-a9d2-968c20a133c3_biasIsr_1123732_39.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/ec19b6d9-8ae6-48b0-a9d2-968c20a133c3_biasIsr_1123732_39.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash ec35d395937333773e850d91d2eb648d parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fe171e1a-60bd-4f9d-af4c-2e20609233c1_biasIsr_1123837_39.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fe171e1a-60bd-4f9d-af4c-2e20609233c1_biasIsr_1123837_39.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash d14bb99264dfa4966899b4dedd9e8f4c parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/69d8f862-8188-4b74-b279-922d09cf3170_biasIsr_1123730_39.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/69d8f862-8188-4b74-b279-922d09cf3170_biasIsr_1123730_39.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 3202c6e58038e8cbbd47dbd22e84a3b0 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6865c946-f89f-4482-b1f3-0703bb5ba244_biasIsr_1123735_39.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6865c946-f89f-4482-b1f3-0703bb5ba244_biasIsr_1123735_39.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 690d7786fc9a4135d9ec413846027dd7 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/09f7d596-548d-4b91-93da-3df144c86f85_biasIsr_1123728_39.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/09f7d596-548d-4b91-93da-3df144c86f85_biasIsr_1123728_39.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash d7e555805eab9a50955e6ec123dfa7e6 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/2454605e-7e74-434a-8810-79c2e6bcb0d5_biasIsr_1123734_39.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/2454605e-7e74-434a-8810-79c2e6bcb0d5_biasIsr_1123734_39.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 1bde46bc0a0d993380117c1f726315cd parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/b508d042-2db6-45a2-bf0d-87f334aa7953_biasIsr_1123832_39.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/b508d042-2db6-45a2-bf0d-87f334aa7953_biasIsr_1123832_39.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 0f365001188d379ff1db115679d5b89a parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/471c89a7-fe2f-46ac-9b6a-82ead4384174_biasIsr_1123833_39.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/471c89a7-fe2f-46ac-9b6a-82ead4384174_biasIsr_1123833_39.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash b7f65c707f6729aecb0b6bb1093d70de parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/35f7b8c9-6fde-4aaf-a4d3-3472c0b10018_biasIsr_1123836_39.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/35f7b8c9-6fde-4aaf-a4d3-3472c0b10018_biasIsr_1123836_39.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash fc98fe16c1f0357194a31d273367e40c parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1154ead-1b39-45fd-af9e-99fb44e28e6b_biasIsr_1123831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1154ead-1b39-45fd-af9e-99fb44e28e6b_biasIsr_1123831_39.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash c6764b33aebf79891bc373fcff58d223 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/cbde7a64-41be-472a-87a7-d7f46ad73139_biasIsr_1123736_39.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/cbde7a64-41be-472a-87a7-d7f46ad73139_biasIsr_1123736_39.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash b03417553dcd7e1bcd2e182220b8bffb parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/01811980-800b-425d-bfbe-d09c9c137d5b_biasIsr_1123829_39.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/01811980-800b-425d-bfbe-d09c9c137d5b_biasIsr_1123829_39.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 2c79331c5f742fc62fcf36ff0e4bcb16 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/b5cb8d06-0e15-494f-a5dd-6414ab7fb5dd_biasIsr_1123738_39.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/b5cb8d06-0e15-494f-a5dd-6414ab7fb5dd_biasIsr_1123738_39.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App cpBiasCombine, waiting on task 437, task 438, task 439, task 440, task 441, task 442, task 443, task 444, task 445, task 446, task 447, task 448, task 449, task 450, task 451, task 452, task 453, task 454, task 455, task 456, task 457, task 458 parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 459 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 740859141c9f3f5a908f419cfa3691c8 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6de3eed4-8093-4782-979e-f2a4626504f2_biasIsr_1123833_18.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6de3eed4-8093-4782-979e-f2a4626504f2_biasIsr_1123833_18.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash b4f67c291123a9fd4274d477105eb488 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/b50c3944-110e-4e32-8bda-3b24610841cf_biasIsr_1123733_18.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/b50c3944-110e-4e32-8bda-3b24610841cf_biasIsr_1123733_18.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 5aa44415366295e5d148e6cd995a3c58 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7699defa-bec8-4068-8cb4-f792829f5a1f_biasIsr_1123738_18.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7699defa-bec8-4068-8cb4-f792829f5a1f_biasIsr_1123738_18.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash f99d154f3e2b112c17483a29fb0b65d6 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/235627a5-c22d-438a-a336-ac3ef6fa4933_biasIsr_1123737_18.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/235627a5-c22d-438a-a336-ac3ef6fa4933_biasIsr_1123737_18.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash edb5922a166c4f2f99897536c5479532 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/50b3c42e-7b00-41af-92ce-9aca5971744c_biasIsr_1123734_18.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/50b3c42e-7b00-41af-92ce-9aca5971744c_biasIsr_1123734_18.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash d68bbcd0c74566dd143e631206c6d791 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d77fbdad-3e30-4a19-aac5-c2445f214b1c_biasIsr_1123832_18.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d77fbdad-3e30-4a19-aac5-c2445f214b1c_biasIsr_1123832_18.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 1916eb10f35d382506a4aec84ef7fc96 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/ccda6970-baa0-4088-9d60-3de7006d68c9_biasIsr_1123829_18.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/ccda6970-baa0-4088-9d60-3de7006d68c9_biasIsr_1123829_18.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash fa4db247202add80b93f052586444f26 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fcebbbf7-aa07-43be-a93e-396b021dfc8a_biasIsr_1123837_18.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fcebbbf7-aa07-43be-a93e-396b021dfc8a_biasIsr_1123837_18.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash fee9ceb57329235a51b98d93b0359f3e parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/fcd199f0-1c18-4132-bc96-952c4aaa323d_biasIsr_1123735_18.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/fcd199f0-1c18-4132-bc96-952c4aaa323d_biasIsr_1123735_18.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 601b56409573ac5689dc149aaa011544 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fdf24e6f-d187-42d5-bde8-bda28d2e338c_biasIsr_1123732_18.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fdf24e6f-d187-42d5-bde8-bda28d2e338c_biasIsr_1123732_18.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 7a02ae5f8e710404524be252eaf72e0c parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/bee9f3c1-d49c-404c-999f-c7791bc2a751_biasIsr_1123728_18.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/bee9f3c1-d49c-404c-999f-c7791bc2a751_biasIsr_1123728_18.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 6dadbe64108bd19003879287f0846a52 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/11140517-089b-44c7-addf-cbdd2b5691a1_biasIsr_1123731_18.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/11140517-089b-44c7-addf-cbdd2b5691a1_biasIsr_1123731_18.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash bb599be568a170eb73075d6595de4e31 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/9e7ff0a9-50b3-42aa-ab24-5b45f8e251f1_biasIsr_1123835_18.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/9e7ff0a9-50b3-42aa-ab24-5b45f8e251f1_biasIsr_1123835_18.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash c47e318ceb7450a30d88151b6302c199 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/a4d5f53d-5051-4519-b12b-25e20296fbfc_biasIsr_1123839_18.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/a4d5f53d-5051-4519-b12b-25e20296fbfc_biasIsr_1123839_18.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 3dd69d2d6d0bb62537630fa68c872fdb parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/82f67338-5f49-4b3b-9b74-7b1a3c58ee4f_biasIsr_1123831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/82f67338-5f49-4b3b-9b74-7b1a3c58ee4f_biasIsr_1123831_18.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash a4ddabf206fd5a03f329f44c051dbe8f parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f895d509-5bfe-4111-8def-d239ce214cdf_biasIsr_1123836_18.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f895d509-5bfe-4111-8def-d239ce214cdf_biasIsr_1123836_18.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash b383dae2c2627a24ea53a8e77ed51453 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e8f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/b7f71534-1b72-4475-9d9d-be5d245c0c09_biasIsr_1123838_18.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/b7f71534-1b72-4475-9d9d-be5d245c0c09_biasIsr_1123838_18.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 6f51e3238f981290877e6a5f3e5fb6a0 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/c1bbd253-c995-4784-8c4e-15d2af07423e_biasIsr_1123830_18.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/c1bbd253-c995-4784-8c4e-15d2af07423e_biasIsr_1123830_18.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash e7b1aaf64f0aaee360a6bce8cc60b9c1 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/690dd582-fdd5-4ab8-87ef-117b6a4df0a8_biasIsr_1123736_18.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/690dd582-fdd5-4ab8-87ef-117b6a4df0a8_biasIsr_1123736_18.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 1521ee83d5ab1f3ffbbd3af5d006c07e parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1255d180-8d39-4fdb-b5cf-759bb204028b_biasIsr_1123730_18.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1255d180-8d39-4fdb-b5cf-759bb204028b_biasIsr_1123730_18.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 8df89b015ab08d05935efa6009ca5247 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/495fa671-49e7-4946-a2ef-185f5bf1985b_biasIsr_1123729_18.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/495fa671-49e7-4946-a2ef-185f5bf1985b_biasIsr_1123729_18.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 936791c3f32103de420058f861db2cb2 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c654dc80-a5f6-4b10-9409-3410beea188c_biasIsr_1123834_18.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c654dc80-a5f6-4b10-9409-3410beea188c_biasIsr_1123834_18.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App cpBiasCombine, waiting on task 460, task 461, task 462, task 463, task 464, task 465, task 466, task 467, task 468, task 469, task 470, task 471, task 472, task 473, task 474, task 475, task 476, task 477, task 478, task 479, task 480, task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash fe3c9ebc43e96ce08edaf67d77c62f4c parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cba20cc5-2aca-4f3e-9696-6803463d6ea7_biasIsr_1123837_54.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cba20cc5-2aca-4f3e-9696-6803463d6ea7_biasIsr_1123837_54.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 10e6f161e9ad7475cc1673974a137833 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/34b46651-3f5b-4412-a2b0-d3ba1ce6a7b3_biasIsr_1123829_54.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/34b46651-3f5b-4412-a2b0-d3ba1ce6a7b3_biasIsr_1123829_54.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash ecd673314fcaff3d6418a575f73a7aee parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/46817e41-3b79-43ee-b595-9eddfff3d5b2_biasIsr_1123730_54.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/46817e41-3b79-43ee-b595-9eddfff3d5b2_biasIsr_1123730_54.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash d3c37d1ffa85554bdb3ad598fd13305c parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/06dacbb7-2744-4324-9f10-bc24c83e5e65_biasIsr_1123735_54.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/06dacbb7-2744-4324-9f10-bc24c83e5e65_biasIsr_1123735_54.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 96e8c29daf057cb68fc6d08f7846f52f parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8cc180ce-164f-4215-9f12-ebcee9d8beff_biasIsr_1123734_54.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8cc180ce-164f-4215-9f12-ebcee9d8beff_biasIsr_1123734_54.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 5faf12bb186693618619925d358d8f72 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8460fc61-ad12-4cf8-8dba-f00d249fa96d_biasIsr_1123728_54.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8460fc61-ad12-4cf8-8dba-f00d249fa96d_biasIsr_1123728_54.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash a5e28b6576586f4a44f0fb5e2e59a001 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/38ce022c-88b2-4257-80d8-a5c366d5cbd9_biasIsr_1123832_54.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/38ce022c-88b2-4257-80d8-a5c366d5cbd9_biasIsr_1123832_54.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 1966de477c5c22f752757ad66b6cc0a5 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/26ee45f8-a0b3-44d6-be30-7bcccafd50ff_biasIsr_1123738_54.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/26ee45f8-a0b3-44d6-be30-7bcccafd50ff_biasIsr_1123738_54.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash ad9937dfb7cf3c291cdf88a49785b88d parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/5cc2f086-67d4-43f1-9c8a-7a963c471277_biasIsr_1123733_54.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/5cc2f086-67d4-43f1-9c8a-7a963c471277_biasIsr_1123733_54.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 89d88e4eda28feb062f61b2dd20abf5f parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/68a90a5c-ba6d-4e0e-bf32-14f3cfa551da_biasIsr_1123736_54.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/68a90a5c-ba6d-4e0e-bf32-14f3cfa551da_biasIsr_1123736_54.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 5d6f964f15626c597d578f60724f2e52 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/789c20c7-9371-4a6c-a82c-534b81501a34_biasIsr_1123834_54.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/789c20c7-9371-4a6c-a82c-534b81501a34_biasIsr_1123834_54.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash a3e6b7d5fd0a078e6f66e7bce3d48356 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4f856bc1-3833-44c5-a43a-1cd797bb9fd8_biasIsr_1123835_54.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4f856bc1-3833-44c5-a43a-1cd797bb9fd8_biasIsr_1123835_54.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash c799b2cf0f9fea0ba52abad60b79edd7 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e2facce1-c897-4c52-88a3-16d62109d9f3_biasIsr_1123836_54.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e2facce1-c897-4c52-88a3-16d62109d9f3_biasIsr_1123836_54.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash a684acd5cc01c61b2b4a4077da5a73d7 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/69fbb9dd-c71f-45f4-aa83-b2f8331ece96_biasIsr_1123732_54.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/69fbb9dd-c71f-45f4-aa83-b2f8331ece96_biasIsr_1123732_54.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 17fc3014b4482566fb708947baa96ca5 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d25b374f-b60f-4efa-923f-8b69b6b9a0d5_biasIsr_1123838_54.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d25b374f-b60f-4efa-923f-8b69b6b9a0d5_biasIsr_1123838_54.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash da5b26fef9debd84692da7409227e337 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6ede98e6-e83a-423e-b1c7-f85bd3a45f3f_biasIsr_1123737_54.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6ede98e6-e83a-423e-b1c7-f85bd3a45f3f_biasIsr_1123737_54.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 3e143b6553a835a63d1be14830122c5d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b13d11e0-d410-4fc5-b572-1e415a815bbc_biasIsr_1123833_54.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b13d11e0-d410-4fc5-b572-1e415a815bbc_biasIsr_1123833_54.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash cb0bf2773aeaeb81ae10606a27c27f94 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/1d558784-d101-4fce-bf5d-f263fb45fdec_biasIsr_1123830_54.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/1d558784-d101-4fce-bf5d-f263fb45fdec_biasIsr_1123830_54.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash a063032a4d595a1860b87bf111fc0c30 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/6d225301-2869-4da4-8d47-2e54ed26f406_biasIsr_1123831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/6d225301-2869-4da4-8d47-2e54ed26f406_biasIsr_1123831_54.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 0d3b5b4a7c461e71c65a94b5f0bf09ab parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b9aebcda-7c74-41e5-8319-85a62ae7c4c5_biasIsr_1123731_54.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b9aebcda-7c74-41e5-8319-85a62ae7c4c5_biasIsr_1123731_54.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 391bdd20e8e4b2204ea8fe1d3a66e84e parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8a33b955-9ce8-4626-8138-72403cf07d8e_biasIsr_1123729_54.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8a33b955-9ce8-4626-8138-72403cf07d8e_biasIsr_1123729_54.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 46751c8a8f5da82101a62e346ebac620 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d7a2b5de-1bc1-470f-9711-e9436abbdae6_biasIsr_1123839_54.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d7a2b5de-1bc1-470f-9711-e9436abbdae6_biasIsr_1123839_54.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App cpBiasCombine, waiting on task 483, task 484, task 485, task 486, task 487, task 488, task 489, task 490, task 491, task 492, task 493, task 494, task 495, task 496, task 497, task 498, task 499, task 500, task 501, task 502, task 503, task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 4c907097ea144a30fa57bb2833a7a52f parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f493439f-d654-41e0-b639-c116fadbcf65_biasIsr_1123830_59.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f493439f-d654-41e0-b639-c116fadbcf65_biasIsr_1123830_59.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 9e4e8a88798f244f8c1d930a840e9a49 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/35dea5ff-4b94-4e1c-afdd-d9cc823157db_biasIsr_1123728_59.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/35dea5ff-4b94-4e1c-afdd-d9cc823157db_biasIsr_1123728_59.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash ed149d35a00565f21de0f75ca3af42fc parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2d23d946-894d-4656-882d-679039dc6ac2_biasIsr_1123730_59.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2d23d946-894d-4656-882d-679039dc6ac2_biasIsr_1123730_59.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 7f45e4cb6fe6ddb0ab8c98e0d8fb1184 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a02f29b1-2795-4a79-80b9-1df595ccfa83_biasIsr_1123835_59.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/a02f29b1-2795-4a79-80b9-1df595ccfa83_biasIsr_1123835_59.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 20bcff2f9743fce445a3b7316edc9acf parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d13c2e3b-06ae-4923-908a-59433e313afd_biasIsr_1123736_59.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d13c2e3b-06ae-4923-908a-59433e313afd_biasIsr_1123736_59.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 37b127ee4bceec735b992c8730013f69 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fa3633fd-6827-4b97-aec6-ec0bfc588141_biasIsr_1123733_59.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fa3633fd-6827-4b97-aec6-ec0bfc588141_biasIsr_1123733_59.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash b35856402184271d31727913d54a6426 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/15a6be9c-806e-4cea-bc65-d7341ef6daf1_biasIsr_1123839_59.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/15a6be9c-806e-4cea-bc65-d7341ef6daf1_biasIsr_1123839_59.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 0fcf27dc92bd75aa06f47ecdfe9356fb parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/76671ce9-195a-47ef-9d90-7d7dc998f4cc_biasIsr_1123832_59.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/76671ce9-195a-47ef-9d90-7d7dc998f4cc_biasIsr_1123832_59.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 2ac9c72aed10697ecac5418933d95195 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/42adbed9-e2a4-475d-91aa-310e4f2e19e4_biasIsr_1123732_59.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/42adbed9-e2a4-475d-91aa-310e4f2e19e4_biasIsr_1123732_59.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash c444be9552ced631f6bde5211074035f parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e9537cb6-634d-4a73-8c42-86d4ef018461_biasIsr_1123734_59.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e9537cb6-634d-4a73-8c42-86d4ef018461_biasIsr_1123734_59.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 8a676fc15aaf82d0c5cd39389048d9c0 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8a575710-1e85-4b07-a83b-3b8c2ca7363b_biasIsr_1123738_59.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8a575710-1e85-4b07-a83b-3b8c2ca7363b_biasIsr_1123738_59.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash f91fe7f336e935a58207eefc79a348f7 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fe213e5d-fd7e-422b-a598-eba75220bcd0_biasIsr_1123837_59.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fe213e5d-fd7e-422b-a598-eba75220bcd0_biasIsr_1123837_59.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash a22bb4172f3a8c1cea1df2281a44d25c parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e18ad939-aecb-432e-8ef9-2676c490288a_biasIsr_1123829_59.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e18ad939-aecb-432e-8ef9-2676c490288a_biasIsr_1123829_59.stderr parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash b96891d4f76a14a408506fff7ac5e88e parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8e79643f-3cd3-407b-b1fa-13aab3e826c8_biasIsr_1123729_59.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8e79643f-3cd3-407b-b1fa-13aab3e826c8_biasIsr_1123729_59.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 5337ed3e30c62d13eabd8344dbbe6d23 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3b4a9cbe-9c94-467c-9909-98039279820c_biasIsr_1123831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3b4a9cbe-9c94-467c-9909-98039279820c_biasIsr_1123831_59.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 64f95e5f5e6a7999cc7d7ce0fdb09984 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d336e4ca-8911-45fd-a204-f562cb599d59_biasIsr_1123737_59.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d336e4ca-8911-45fd-a204-f562cb599d59_biasIsr_1123737_59.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 939733b305ea1b241ba5e9a11e965923 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d36e8910-2034-47f8-91ed-699592a5f08c_biasIsr_1123735_59.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d36e8910-2034-47f8-91ed-699592a5f08c_biasIsr_1123735_59.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash da840f5d804833e0c9a0b69a93d02e4c parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/47b8f278-0014-4373-bd24-b02566a687b6_biasIsr_1123838_59.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/47b8f278-0014-4373-bd24-b02566a687b6_biasIsr_1123838_59.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash b5fef93e8a0fed92a260016724220241 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/166c7356-6698-426d-86d1-c6ce2a46f904_biasIsr_1123834_59.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/166c7356-6698-426d-86d1-c6ce2a46f904_biasIsr_1123834_59.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash fa7219d962645fb461595b95433a29cc parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a3057656-a34f-4115-a4a3-4541fb385914_biasIsr_1123731_59.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a3057656-a34f-4115-a4a3-4541fb385914_biasIsr_1123731_59.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 47220672c7aabfc4df3e3c3d4f81df14 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/41bda2c1-27d2-48f4-90e2-cb4e0385c2d7_biasIsr_1123833_59.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/41bda2c1-27d2-48f4-90e2-cb4e0385c2d7_biasIsr_1123833_59.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 3472f7659f1aac24d73847665a3bd97e parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ec7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1fde0bf4-8409-4867-bf81-613caa3d954d_biasIsr_1123836_59.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1fde0bf4-8409-4867-bf81-613caa3d954d_biasIsr_1123836_59.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App cpBiasCombine, waiting on task 506, task 507, task 508, task 509, task 510, task 511, task 512, task 513, task 514, task 515, task 516, task 517, task 518, task 519, task 520, task 521, task 522, task 523, task 524, task 525, task 526, task 527 parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 528 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash d99b32aa1524fc2d158090594fc04406 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/90bf89d9-1415-4a8c-8042-e61b6b2fca27_biasIsr_1123732_41.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/90bf89d9-1415-4a8c-8042-e61b6b2fca27_biasIsr_1123732_41.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 22b0c36b1d8385f5f0770844e61d7f39 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7e57f07a-d065-44e4-9f8a-d50824f199f1_biasIsr_1123738_41.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7e57f07a-d065-44e4-9f8a-d50824f199f1_biasIsr_1123738_41.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash d0f341333ba7833ea65df166a6502e9c parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/057fce92-7fec-4743-b026-16e6c887aae4_biasIsr_1123834_41.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/057fce92-7fec-4743-b026-16e6c887aae4_biasIsr_1123834_41.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 748fbc7b61f6eae7b6ba9bf8a2d58062 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/10e5c3a9-f1b5-42ef-9f63-4364a237843c_biasIsr_1123832_41.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/10e5c3a9-f1b5-42ef-9f63-4364a237843c_biasIsr_1123832_41.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 13e429aeb43c62c031b113f824cd999b parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/82e38c22-0e8d-4492-a9c2-0529463754ef_biasIsr_1123836_41.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/82e38c22-0e8d-4492-a9c2-0529463754ef_biasIsr_1123836_41.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 874a4942419ead0beb4e810eb7eefac2 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/69200d55-e91f-4744-a5ef-977c8737b823_biasIsr_1123734_41.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/69200d55-e91f-4744-a5ef-977c8737b823_biasIsr_1123734_41.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash a4f8b7022a243c1d6c956dfdf4b59622 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/0b95b034-c331-4199-8cd5-4da744082c7d_biasIsr_1123838_41.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/0b95b034-c331-4199-8cd5-4da744082c7d_biasIsr_1123838_41.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash d1f76732ef88785442370947324d8cee parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/69b710cf-acdf-4f55-9662-8e0ab4342a79_biasIsr_1123835_41.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/69b710cf-acdf-4f55-9662-8e0ab4342a79_biasIsr_1123835_41.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash e37f48e2e3a68b3e31e716ec8472c70f parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/919bda5d-e393-4a68-b6ea-8d2bc4eaa803_biasIsr_1123833_41.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/919bda5d-e393-4a68-b6ea-8d2bc4eaa803_biasIsr_1123833_41.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 366c8bbe373b7a8eb52dd0da82e3bb98 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/fcceef27-426c-495d-9ce6-774f908863c9_biasIsr_1123831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/fcceef27-426c-495d-9ce6-774f908863c9_biasIsr_1123831_41.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 049b34f5a4ca9294ccd3c83887b72919 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/3f5f986e-d443-4182-bf20-f867b5271f8f_biasIsr_1123829_41.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/3f5f986e-d443-4182-bf20-f867b5271f8f_biasIsr_1123829_41.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 94e2753a5b63518f9cf985e6cc7c5bbd parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cf7f6216-7cfe-4311-8e59-de7fb18edfcb_biasIsr_1123837_41.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cf7f6216-7cfe-4311-8e59-de7fb18edfcb_biasIsr_1123837_41.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash a17e271636849cd5b1f35481a987a0be parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/7fd8024d-4365-4fa0-821f-3c6a20b0f1a0_biasIsr_1123728_41.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/7fd8024d-4365-4fa0-821f-3c6a20b0f1a0_biasIsr_1123728_41.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash da0adc2de50980167549c35159c8825b parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/672ab8c7-5237-485a-a42a-6b6c11d36306_biasIsr_1123736_41.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/672ab8c7-5237-485a-a42a-6b6c11d36306_biasIsr_1123736_41.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash dd004da8c801501c10a7252e2c680d04 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/88a5122f-6e57-40ce-9484-4f17a5f81317_biasIsr_1123729_41.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/88a5122f-6e57-40ce-9484-4f17a5f81317_biasIsr_1123729_41.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 4002971c35f1830517ed660b12905c12 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/e5bde4a7-850f-4620-9571-d8418ac838d1_biasIsr_1123731_41.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/e5bde4a7-850f-4620-9571-d8418ac838d1_biasIsr_1123731_41.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 003071a0bbc917f42c2c868b25c27f0f parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/645a8a1a-29fd-4377-9012-5dc462f6dd30_biasIsr_1123830_41.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/645a8a1a-29fd-4377-9012-5dc462f6dd30_biasIsr_1123830_41.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 64c663a5c6a0ddfe29c5d592b97538cd parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d7c989bb-7662-4fdb-b554-6532bf53393c_biasIsr_1123735_41.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d7c989bb-7662-4fdb-b554-6532bf53393c_biasIsr_1123735_41.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 591bbbfd97de9f8022cadc1dd5defa2e parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/54799f04-7311-48fc-affa-e0d527999c6d_biasIsr_1123730_41.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/54799f04-7311-48fc-affa-e0d527999c6d_biasIsr_1123730_41.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash b1fb9195f2970c821643fcebed8e0714 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0c7b7492-7132-4010-8010-dd3f0e96141a_biasIsr_1123737_41.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0c7b7492-7132-4010-8010-dd3f0e96141a_biasIsr_1123737_41.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 1de0c33717d0c93a00de18a1240f27e9 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/33dd1188-4e98-407e-bb7a-e3d0b4097733_biasIsr_1123839_41.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/33dd1188-4e98-407e-bb7a-e3d0b4097733_biasIsr_1123839_41.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash e0ef87671b3bbea0156c6c9520b0d277 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/9aa4801d-e72c-4aed-b8ca-94be2878adcd_biasIsr_1123733_41.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/9aa4801d-e72c-4aed-b8ca-94be2878adcd_biasIsr_1123733_41.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App cpBiasCombine, waiting on task 529, task 530, task 531, task 532, task 533, task 534, task 535, task 536, task 537, task 538, task 539, task 540, task 541, task 542, task 543, task 544, task 545, task 546, task 547, task 548, task 549, task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash f139f1cbdd69122c5eed1f25c1b17721 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/b267c196-4dc5-4e62-8d64-a9c4b4cfba0c_biasIsr_1123838_24.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/b267c196-4dc5-4e62-8d64-a9c4b4cfba0c_biasIsr_1123838_24.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 1a646fdebfd32880728f3607fecd4d6d parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/052da4eb-2b50-466a-87e6-9d966160c210_biasIsr_1123735_24.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/052da4eb-2b50-466a-87e6-9d966160c210_biasIsr_1123735_24.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 7664efb1c5c3ba11be03e48ff021855f parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c4d7cd8c-4dd1-4ae9-8679-7a6ff5e0e2b6_biasIsr_1123835_24.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c4d7cd8c-4dd1-4ae9-8679-7a6ff5e0e2b6_biasIsr_1123835_24.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash bc74e99cfe4d2b43936787ba3162b1c8 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/aedb68cd-512f-4b34-a292-c527c82344f5_biasIsr_1123731_24.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/aedb68cd-512f-4b34-a292-c527c82344f5_biasIsr_1123731_24.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash c8312ec924bd9f6153a80dd789b77825 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3a369b27-2237-4977-95a8-255c4b5d23dd_biasIsr_1123831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3a369b27-2237-4977-95a8-255c4b5d23dd_biasIsr_1123831_24.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 49fcb152426cd9cb3f6764031d7bf984 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/940d1884-ff7d-4246-9bb6-6534f947b942_biasIsr_1123733_24.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/940d1884-ff7d-4246-9bb6-6534f947b942_biasIsr_1123733_24.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash fc80f3c1d1ad44ff1b5bc333a8611f9a parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/71e28bb0-25b0-4faa-b36c-28bbe55500c4_biasIsr_1123728_24.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/71e28bb0-25b0-4faa-b36c-28bbe55500c4_biasIsr_1123728_24.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 2dde9e1f6d4d1961a196cff534569554 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/18b57c3d-4264-401e-bfc2-4c0ed60895b7_biasIsr_1123834_24.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/18b57c3d-4264-401e-bfc2-4c0ed60895b7_biasIsr_1123834_24.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 55a2ecbc22842b5a0c190583908517a9 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/aad4ed4b-908b-453a-b9ed-baa20cb9265f_biasIsr_1123837_24.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/aad4ed4b-908b-453a-b9ed-baa20cb9265f_biasIsr_1123837_24.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 9470cad843e76b15a1517bf412bab8b1 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/8873e9de-db4e-4a46-b11f-1cf1b23e39b8_biasIsr_1123830_24.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/8873e9de-db4e-4a46-b11f-1cf1b23e39b8_biasIsr_1123830_24.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 2b4a3c88e9a835d163bd04340b73f747 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b42a9d96-f32d-4600-b782-124baeed2be2_biasIsr_1123839_24.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b42a9d96-f32d-4600-b782-124baeed2be2_biasIsr_1123839_24.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 9b42a8ff004dc0f100bcb975aff3105b parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/580f744d-ef31-4956-a7da-9f7c91f1cdcd_biasIsr_1123833_24.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/580f744d-ef31-4956-a7da-9f7c91f1cdcd_biasIsr_1123833_24.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 38d095e501d90f16cec53145a2ca2743 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e0a988ca-81fb-41bb-8ead-bbade604f859_biasIsr_1123829_24.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e0a988ca-81fb-41bb-8ead-bbade604f859_biasIsr_1123829_24.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 06a88837207fe5b87e2c72a227a53abc parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/0034e787-e6fc-44aa-aa75-15e4d634d5f5_biasIsr_1123729_24.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/0034e787-e6fc-44aa-aa75-15e4d634d5f5_biasIsr_1123729_24.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 0112e51318f041fe5d45e8dae49e329a parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/35cf9f62-89ea-4098-a876-006ee8ff457d_biasIsr_1123732_24.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/35cf9f62-89ea-4098-a876-006ee8ff457d_biasIsr_1123732_24.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash fb63162999397570ee53fb8a70187a00 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/99ae7613-f6ca-427c-87ac-f445ed822616_biasIsr_1123737_24.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/99ae7613-f6ca-427c-87ac-f445ed822616_biasIsr_1123737_24.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 773d9152d2ad2d369490f5a1d2e72fc7 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/28195851-80f2-49e2-819d-18ac0004c196_biasIsr_1123832_24.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/28195851-80f2-49e2-819d-18ac0004c196_biasIsr_1123832_24.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash c6ad8a6c95d59bbc2a8b0ea1aee79956 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/811bfd8e-8784-4d53-948a-196c1706c28f_biasIsr_1123730_24.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/811bfd8e-8784-4d53-948a-196c1706c28f_biasIsr_1123730_24.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 6bdd8fb569359e2d9ed5ebe63188fcd2 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/60cbada3-e70e-49d5-b900-5583ab85c9ae_biasIsr_1123836_24.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/60cbada3-e70e-49d5-b900-5583ab85c9ae_biasIsr_1123836_24.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash a6ddf8f8501ca6e133e335d9f30365a0 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8b75d5e0-6b06-409b-974d-c326b9ad2237_biasIsr_1123734_24.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8b75d5e0-6b06-409b-974d-c326b9ad2237_biasIsr_1123734_24.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 58d84ffa79a85b3e3ef18dc086219108 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f62c3493-412c-426c-843c-bb77e0c69d02_biasIsr_1123738_24.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f62c3493-412c-426c-843c-bb77e0c69d02_biasIsr_1123738_24.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 115159826bd50901046938043f5710f3 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ef7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a7ca0f73-3ace-488b-a85d-6317981d0576_biasIsr_1123736_24.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a7ca0f73-3ace-488b-a85d-6317981d0576_biasIsr_1123736_24.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App cpBiasCombine, waiting on task 552, task 553, task 554, task 555, task 556, task 557, task 558, task 559, task 560, task 561, task 562, task 563, task 564, task 565, task 566, task 567, task 568, task 569, task 570, task 571, task 572, task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 9bd4aab009a71172b9e9f384b92ea878 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/07000119-b1f2-4275-92ae-5f9284cfd433_biasIsr_1123735_35.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/07000119-b1f2-4275-92ae-5f9284cfd433_biasIsr_1123735_35.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 34d0e30ef1258807c349fc811f2ea487 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c8400733-f665-46b5-bf16-df9356a5e327_biasIsr_1123837_35.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c8400733-f665-46b5-bf16-df9356a5e327_biasIsr_1123837_35.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash c6d3c87ee3f589c41a9c6eb039afc752 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/5e9101dd-8dbb-4c18-8adc-385e6b608fed_biasIsr_1123730_35.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/5e9101dd-8dbb-4c18-8adc-385e6b608fed_biasIsr_1123730_35.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash ebc241554b4935ed2db0fc03de0993a2 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f94a681c-ee3d-4615-8607-32c4817f88f8_biasIsr_1123738_35.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f94a681c-ee3d-4615-8607-32c4817f88f8_biasIsr_1123738_35.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash c768373af1067916a14ca61d62102ad9 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/7e26efc0-3928-477a-ae96-132db6764b81_biasIsr_1123836_35.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/7e26efc0-3928-477a-ae96-132db6764b81_biasIsr_1123836_35.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 2dffedb672d1f4f3e6a12667e6a739d3 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5838979d-8bd5-4a52-af8f-85dfeed725ac_biasIsr_1123829_35.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5838979d-8bd5-4a52-af8f-85dfeed725ac_biasIsr_1123829_35.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 33c697bfb5f2967c4604d876cb2d0433 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/000471a6-1a49-483d-bcfd-7152e8a3f819_biasIsr_1123736_35.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/000471a6-1a49-483d-bcfd-7152e8a3f819_biasIsr_1123736_35.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 4bd0a067138820647ef139704969a620 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8da8a1c6-8aac-492f-89b3-903f34e86635_biasIsr_1123728_35.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8da8a1c6-8aac-492f-89b3-903f34e86635_biasIsr_1123728_35.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 9d5741e0da9ed4c68062c5df983f83d8 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b1a11b20-c870-464c-bd42-aa867506cb84_biasIsr_1123833_35.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b1a11b20-c870-464c-bd42-aa867506cb84_biasIsr_1123833_35.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash c6af8ad16c644dd50e89e03597a971b2 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/a3ee371e-2fb3-424d-99a4-423df4c80c3a_biasIsr_1123830_35.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/a3ee371e-2fb3-424d-99a4-423df4c80c3a_biasIsr_1123830_35.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 0616bd5363a329788d44b7c976ee0c3e parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f74e67d9-5421-491b-8a63-0abd21c49ab9_biasIsr_1123732_35.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f74e67d9-5421-491b-8a63-0abd21c49ab9_biasIsr_1123732_35.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash ac02e2aa4700e48c64c678746d07ad6a parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/0b01e735-d910-46df-a978-1e53e0574c58_biasIsr_1123839_35.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/0b01e735-d910-46df-a978-1e53e0574c58_biasIsr_1123839_35.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash eda30c56ee96a15789a1fc1354724c35 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/0692ef70-1287-43ac-9d95-b11cc457f2dc_biasIsr_1123835_35.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/0692ef70-1287-43ac-9d95-b11cc457f2dc_biasIsr_1123835_35.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 995e7e3bf82d0a1b80c21d4dbc4c1bed parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c0f4d826-6cd8-4169-9728-2dde8bbe195c_biasIsr_1123731_35.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c0f4d826-6cd8-4169-9728-2dde8bbe195c_biasIsr_1123731_35.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 883ae5a2cb13cc38582886b3f312c447 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/38a6026a-e2e3-471e-afd2-f0a446aca3af_biasIsr_1123838_35.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/38a6026a-e2e3-471e-afd2-f0a446aca3af_biasIsr_1123838_35.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 92558da3a5ba428377186746c26572e1 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/0587332e-d716-463b-be2d-5e4b85779088_biasIsr_1123729_35.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/0587332e-d716-463b-be2d-5e4b85779088_biasIsr_1123729_35.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 082b9a4eb58182ac3cf322755987b2ff parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/d0200938-6b2d-46e4-8608-88adfa4261cb_biasIsr_1123733_35.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/d0200938-6b2d-46e4-8608-88adfa4261cb_biasIsr_1123733_35.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 4e93ef48750d42b9812266af9ad44991 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/a5a79ea8-9b53-4359-8ec1-fb4daefea23a_biasIsr_1123834_35.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/a5a79ea8-9b53-4359-8ec1-fb4daefea23a_biasIsr_1123834_35.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 3ae19dc3389ee6da3a9d7b2d9a2a1622 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/fe38ccc3-8530-44b0-aa4d-a7e71ef2c700_biasIsr_1123734_35.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/fe38ccc3-8530-44b0-aa4d-a7e71ef2c700_biasIsr_1123734_35.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 4eb97df82565ae50120479572abe96ee parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/0b41f4d7-48b5-406d-a979-5d74f63b5b70_biasIsr_1123832_35.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/0b41f4d7-48b5-406d-a979-5d74f63b5b70_biasIsr_1123832_35.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash de8b70304773593c829d8d072e41f374 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/71f71abe-9930-4bb2-963c-6d57ebccef35_biasIsr_1123831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/71f71abe-9930-4bb2-963c-6d57ebccef35_biasIsr_1123831_35.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 60a5fa6c1ca41d4ebc8d9c302edfb464 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/621f2e51-3dab-4589-882f-ca1aecf66edb_biasIsr_1123737_35.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/621f2e51-3dab-4589-882f-ca1aecf66edb_biasIsr_1123737_35.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App cpBiasCombine, waiting on task 575, task 576, task 577, task 578, task 579, task 580, task 581, task 582, task 583, task 584, task 585, task 586, task 587, task 588, task 589, task 590, task 591, task 592, task 593, task 594, task 595, task 596 parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 597 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash dc6b97b78963d64a7490d04f861ec17a parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0eb277e8-098e-4ce2-822f-6b6ba6faeece_biasIsr_1123732_10.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0eb277e8-098e-4ce2-822f-6b6ba6faeece_biasIsr_1123732_10.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 69b00db5914b5105aba34dd85e323849 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/7984dbac-91c7-4b45-9df3-4304d10cb3e6_biasIsr_1123838_10.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/7984dbac-91c7-4b45-9df3-4304d10cb3e6_biasIsr_1123838_10.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash b6b64142d4c9846d4cb0d6980da568b9 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/416a7045-6468-473c-b7fb-ccadbe33b630_biasIsr_1123835_10.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/416a7045-6468-473c-b7fb-ccadbe33b630_biasIsr_1123835_10.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 85b68df5c7742ea4d28b8ce1a63dba9c parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/41b44150-90d1-4d65-b3e2-ec682695e863_biasIsr_1123737_10.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/41b44150-90d1-4d65-b3e2-ec682695e863_biasIsr_1123737_10.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 29b8c955a0c10b05d5ca4b0bb66bbf67 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1b8d6943-3d40-4571-a1f1-97ec6dbef8f5_biasIsr_1123831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1b8d6943-3d40-4571-a1f1-97ec6dbef8f5_biasIsr_1123831_10.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 71f7798c67c8f95b671ab37e02158304 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/4764fae3-0895-4f54-a30c-a5ebb6d5fffb_biasIsr_1123734_10.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/4764fae3-0895-4f54-a30c-a5ebb6d5fffb_biasIsr_1123734_10.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 55ec0970bdb4898fde84946dfc3e7de6 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1812c307-7551-4180-b8d8-1b4802a342fb_biasIsr_1123738_10.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1812c307-7551-4180-b8d8-1b4802a342fb_biasIsr_1123738_10.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash a134a231169ac8134302f3e6b11f16ec parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/720e715b-afff-4bab-9e4d-8ce26515016f_biasIsr_1123731_10.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/720e715b-afff-4bab-9e4d-8ce26515016f_biasIsr_1123731_10.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 014e939b3f6ae9a38312f9e293629d29 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/d70f4159-2cf0-4a4f-b53c-0d0cebab65a9_biasIsr_1123730_10.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/d70f4159-2cf0-4a4f-b53c-0d0cebab65a9_biasIsr_1123730_10.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash aededbf5094d77cf47b4be6577ac4d7e parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/85d58335-393f-4e0f-bc2e-e38e52c132c3_biasIsr_1123735_10.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/85d58335-393f-4e0f-bc2e-e38e52c132c3_biasIsr_1123735_10.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 0863f08239a7d9d486f639fe812eb53b parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/61767821-7eb3-4897-8256-ba9e463f42d6_biasIsr_1123832_10.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/61767821-7eb3-4897-8256-ba9e463f42d6_biasIsr_1123832_10.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 782953948d0e5208667bb03f0c4fdbfe parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6fbb2ddd-a29a-4b6c-bfac-6fcaabf77404_biasIsr_1123834_10.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6fbb2ddd-a29a-4b6c-bfac-6fcaabf77404_biasIsr_1123834_10.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 8263ac3d39befbdd7deb0e3efc0f9b90 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/a5167c1c-78e2-4390-98f2-3ae48a1fd76e_biasIsr_1123829_10.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/a5167c1c-78e2-4390-98f2-3ae48a1fd76e_biasIsr_1123829_10.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 0fb708f7bef8685f8edaab5be5318eb0 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/689afd45-89aa-4ea8-9162-1f063268570c_biasIsr_1123733_10.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/689afd45-89aa-4ea8-9162-1f063268570c_biasIsr_1123733_10.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash dc541f3bec704af95d23a3ac14edb51c parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/5bc0dc03-a6f9-45ce-a2c2-980f273089f3_biasIsr_1123837_10.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/5bc0dc03-a6f9-45ce-a2c2-980f273089f3_biasIsr_1123837_10.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash e8880798f9b687b3b818ceb2a4adb221 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2a120156-55f5-4e41-aab7-a1aeb73d3550_biasIsr_1123728_10.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2a120156-55f5-4e41-aab7-a1aeb73d3550_biasIsr_1123728_10.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 45cd0b39f165b7bcd8fee6b56edf251a parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e3ea4590-26c4-4ff5-9052-eddf5afc430f_biasIsr_1123836_10.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e3ea4590-26c4-4ff5-9052-eddf5afc430f_biasIsr_1123836_10.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 7a8da3838474acc0701b9467f0cd212d parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b4b63c3c-52d9-494d-9c71-f6402de9e4b0_biasIsr_1123830_10.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/b4b63c3c-52d9-494d-9c71-f6402de9e4b0_biasIsr_1123830_10.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 7ea0813fe9a04addcb9a51e04766be55 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b9a58bd7-f0f8-41aa-a3af-4c860f25db89_biasIsr_1123839_10.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b9a58bd7-f0f8-41aa-a3af-4c860f25db89_biasIsr_1123839_10.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 6120f025726576f487ce710f78d69953 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/698098ab-94b6-4a59-a3b4-710cd104bc82_biasIsr_1123833_10.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/698098ab-94b6-4a59-a3b4-710cd104bc82_biasIsr_1123833_10.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash e5680f2a04c0b4e5072d90bd9b31c9e0 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/8242e38b-5a57-4364-baad-a89e6d3576ae_biasIsr_1123736_10.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/8242e38b-5a57-4364-baad-a89e6d3576ae_biasIsr_1123736_10.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 752fad5556f1cf05f777b4b83de5ecbc parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8e8677fe-fce7-4a64-8e91-924a90645ff0_biasIsr_1123729_10.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/8e8677fe-fce7-4a64-8e91-924a90645ff0_biasIsr_1123729_10.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App cpBiasCombine, waiting on task 598, task 599, task 600, task 601, task 602, task 603, task 604, task 605, task 606, task 607, task 608, task 609, task 610, task 611, task 612, task 613, task 614, task 615, task 616, task 617, task 618, task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash f662494ab9446c54b3313ee6b1bf88b4 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e6bd8104-fc71-42f6-af4d-a0294a7ab3fe_biasIsr_1123734_34.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e6bd8104-fc71-42f6-af4d-a0294a7ab3fe_biasIsr_1123734_34.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash c49992e71e489e62ac850e82b5499e8f parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1bba123d-cbfe-4734-bc80-4fdb3cdcd4b6_biasIsr_1123836_34.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1bba123d-cbfe-4734-bc80-4fdb3cdcd4b6_biasIsr_1123836_34.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash e36b2adc5b6833b64f8783a91c29ad84 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/1382c7b6-477d-440a-b184-fe1c85497e5b_biasIsr_1123736_34.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/1382c7b6-477d-440a-b184-fe1c85497e5b_biasIsr_1123736_34.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 1b57f3b79f6a195e36def813ef4c1c9d parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/eea6db47-1006-4e53-807d-5a253458a340_biasIsr_1123735_34.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/eea6db47-1006-4e53-807d-5a253458a340_biasIsr_1123735_34.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 04833f5766ef5031779f52084d689185 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f2d8cf81-7a45-46e9-b71e-c4a3a734c049_biasIsr_1123830_34.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f2d8cf81-7a45-46e9-b71e-c4a3a734c049_biasIsr_1123830_34.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash ce3fdef01bdef3f7d62041308ad14ec3 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8d6b4e57-3198-4321-8b4d-3f3699d4394a_biasIsr_1123738_34.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8d6b4e57-3198-4321-8b4d-3f3699d4394a_biasIsr_1123738_34.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 900223f36b6ed2a8aa94036d70fcc799 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/10b4d45d-127c-461b-a188-833ea8db2e17_biasIsr_1123834_34.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/10b4d45d-127c-461b-a188-833ea8db2e17_biasIsr_1123834_34.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 31e9e1c61dac6578382752348171e60b parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f1fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d13e0ca8-b48f-4820-a615-fa53381053df_biasIsr_1123728_34.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d13e0ca8-b48f-4820-a615-fa53381053df_biasIsr_1123728_34.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 3ad45bdbf5a5113fc6b5903d698e3a8b parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/c0e93be5-d008-464a-9567-13d145d8841c_biasIsr_1123733_34.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/c0e93be5-d008-464a-9567-13d145d8841c_biasIsr_1123733_34.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 880edad38d327e66a38121ba68f54dbe parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb3a88c8-9d62-4c0b-aa68-23011adb47ef_biasIsr_1123839_34.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb3a88c8-9d62-4c0b-aa68-23011adb47ef_biasIsr_1123839_34.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 8e35fc4cf08d0cb0dc52f6ac7ffaf989 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/ffc375b4-c016-4000-b383-bd6c6332fc71_biasIsr_1123832_34.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/ffc375b4-c016-4000-b383-bd6c6332fc71_biasIsr_1123832_34.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 735abeff21bd3bb3294eec54f21ca939 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f502c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c41cdfdd-44bf-42b6-a2fd-8e05c08970b0_biasIsr_1123729_34.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c41cdfdd-44bf-42b6-a2fd-8e05c08970b0_biasIsr_1123729_34.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 996a35624993570587df53b7c592a0d5 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6219eae7-bea1-4dd3-8070-c2d28fd63567_biasIsr_1123838_34.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6219eae7-bea1-4dd3-8070-c2d28fd63567_biasIsr_1123838_34.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 272ad66e450565aed2fce701f3669d31 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1a1356fb-a358-49c6-a8e9-f8bc99f7526a_biasIsr_1123831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1a1356fb-a358-49c6-a8e9-f8bc99f7526a_biasIsr_1123831_34.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash d2ddd3a8fbb6f809d24b58162dd546ce parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/987e422a-efdd-4eee-b294-b6a34beed5b3_biasIsr_1123833_34.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/987e422a-efdd-4eee-b294-b6a34beed5b3_biasIsr_1123833_34.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 3dbacb614f96439a12d3eb36327c9ea5 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2d9a978b-67d7-4b26-8d9c-63786e2b17bb_biasIsr_1123730_34.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2d9a978b-67d7-4b26-8d9c-63786e2b17bb_biasIsr_1123730_34.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash f5528db8e6470b8576ac3dbab938b206 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f505e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/23b5319f-af32-430e-b766-dd8e1233f38d_biasIsr_1123732_34.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/23b5319f-af32-430e-b766-dd8e1233f38d_biasIsr_1123732_34.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash c523fd7457e446cd60f25324d3e6f2c1 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/06e32807-3991-47f2-a759-5d97c335cdd1_biasIsr_1123829_34.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/06e32807-3991-47f2-a759-5d97c335cdd1_biasIsr_1123829_34.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash affff7ebff73c94e0ea4c17e8874680a parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/53eace4f-5ec1-4225-9176-adb34bb5a3da_biasIsr_1123837_34.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/53eace4f-5ec1-4225-9176-adb34bb5a3da_biasIsr_1123837_34.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 392cef694425d2aeecb3922f52bf163b parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/19c4e57b-85f1-4b7d-9b4a-fdea3afc5600_biasIsr_1123835_34.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/19c4e57b-85f1-4b7d-9b4a-fdea3afc5600_biasIsr_1123835_34.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash f7e8420a8baaa399e8666fd4ecceac52 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/bf449944-5a6e-40d7-a3a9-99d263862a8c_biasIsr_1123737_34.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/bf449944-5a6e-40d7-a3a9-99d263862a8c_biasIsr_1123737_34.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 61ee93cc347103421a24b90d9891ad1a parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ed2fde99-bf6b-457a-934b-62da5a0dd24d_biasIsr_1123731_34.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ed2fde99-bf6b-457a-934b-62da5a0dd24d_biasIsr_1123731_34.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App cpBiasCombine, waiting on task 621, task 622, task 623, task 624, task 625, task 626, task 627, task 628, task 629, task 630, task 631, task 632, task 633, task 634, task 635, task 636, task 637, task 638, task 639, task 640, task 641, task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 4009e1e798cf4be0634becce5c1b4d45 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/59974908-9a33-4f3f-a1b8-4020b806685b_biasIsr_1123734_12.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/59974908-9a33-4f3f-a1b8-4020b806685b_biasIsr_1123734_12.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash ae3cb5a0e67f48f3e153b9cc6f637bc6 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/0ce11ebf-fb82-4944-afe8-4e8cb31fc85b_biasIsr_1123836_12.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/0ce11ebf-fb82-4944-afe8-4e8cb31fc85b_biasIsr_1123836_12.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 4ce6e204ffae134cc604a31f7bb19da8 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c163ec80-634b-4873-992f-ca6019b85f3d_biasIsr_1123732_12.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c163ec80-634b-4873-992f-ca6019b85f3d_biasIsr_1123732_12.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash e5e686fd8a30294b85fbcc8583192e4f parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f50ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/991bbda9-9a36-4921-b67f-28167a9628e3_biasIsr_1123728_12.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/991bbda9-9a36-4921-b67f-28167a9628e3_biasIsr_1123728_12.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 87625f9d1b0a1e5d9508b24adb22836e parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1af61ccd-19e3-4d8c-b688-24f4e6140bd7_biasIsr_1123838_12.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1af61ccd-19e3-4d8c-b688-24f4e6140bd7_biasIsr_1123838_12.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 7e9bcee606ad0ec8764b9f20235e7f3a parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/145b488c-4788-4fe4-924c-ceb178b3a436_biasIsr_1123735_12.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/145b488c-4788-4fe4-924c-ceb178b3a436_biasIsr_1123735_12.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 411440334ff7c82000740bc701728545 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a3cc5a89-22b4-4911-865c-37b2bf4f45af_biasIsr_1123730_12.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a3cc5a89-22b4-4911-865c-37b2bf4f45af_biasIsr_1123730_12.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 405b3f855a15bc4f5ccea6e6156abfc3 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/48dfbc31-8480-4e60-85e8-1577403bcae4_biasIsr_1123835_12.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/48dfbc31-8480-4e60-85e8-1577403bcae4_biasIsr_1123835_12.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 6ff7ea4b4a1355765bdfd82d6def1d5d parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/5c44f1aa-7d67-4615-aa51-19dd6b4d8c04_biasIsr_1123737_12.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/5c44f1aa-7d67-4615-aa51-19dd6b4d8c04_biasIsr_1123737_12.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash f9b530ec71a87bd8d42b73fefaab8fc2 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/5eb5e5aa-acfd-47af-b461-051812d83afb_biasIsr_1123839_12.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/5eb5e5aa-acfd-47af-b461-051812d83afb_biasIsr_1123839_12.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 9f3672d6f11420fdd331ec3a8bfc2d55 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/bc90ddc6-7982-491e-9894-eba4946e631f_biasIsr_1123834_12.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/bc90ddc6-7982-491e-9894-eba4946e631f_biasIsr_1123834_12.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash a3a327f5cafed7ebfb79fa2d69eddbdb parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d8038e14-35af-4cd3-907d-682a41c2b1cb_biasIsr_1123736_12.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d8038e14-35af-4cd3-907d-682a41c2b1cb_biasIsr_1123736_12.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 0705e6f5110facd66999228de6c89be5 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/94ddd0f8-1a30-4b47-92a0-54469718202f_biasIsr_1123832_12.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/94ddd0f8-1a30-4b47-92a0-54469718202f_biasIsr_1123832_12.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 010856a875229a2a4f6665dd66a0de06 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f51f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/5e026402-2813-40a2-964d-f282f82de8c3_biasIsr_1123733_12.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/5e026402-2813-40a2-964d-f282f82de8c3_biasIsr_1123733_12.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 747f19002b309472597921fb65d709b9 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e5cc939d-a328-4076-9c0f-16ab76587213_biasIsr_1123738_12.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/e5cc939d-a328-4076-9c0f-16ab76587213_biasIsr_1123738_12.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 9e5b35b8f5567d4e8470113d6e7c5693 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f520c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3e4057db-ca8e-49ec-abd7-97e6eb987c8e_biasIsr_1123731_12.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3e4057db-ca8e-49ec-abd7-97e6eb987c8e_biasIsr_1123731_12.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 9c61956aff5b425870f457e390e99ec7 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/705aa1ed-18d6-4e1d-b697-50aea0163647_biasIsr_1123837_12.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/705aa1ed-18d6-4e1d-b697-50aea0163647_biasIsr_1123837_12.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash e7629103b1d9abc3f3827517c6948513 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1ea28fc-f83a-4de6-86ae-08ec58c83741_biasIsr_1123831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1ea28fc-f83a-4de6-86ae-08ec58c83741_biasIsr_1123831_12.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 9ff608b01d88e8bd74d0e92b28e2d91a parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f522a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/68cafff3-dc83-41a2-84da-d8125bed16b8_biasIsr_1123830_12.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/68cafff3-dc83-41a2-84da-d8125bed16b8_biasIsr_1123830_12.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash e01bfce8bd03cf50a7a0c7b7ce3b3a36 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/caa9311b-71b3-4aee-9e8b-1b9d83a72762_biasIsr_1123833_12.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/caa9311b-71b3-4aee-9e8b-1b9d83a72762_biasIsr_1123833_12.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash f51f86b0cfe4ca6a84837ab6e859492d parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f523e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/eaa34bca-c514-46bb-9379-2a82602f98b1_biasIsr_1123829_12.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/eaa34bca-c514-46bb-9379-2a82602f98b1_biasIsr_1123829_12.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 794552a6e5c4044b78c035cda08bb53a parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b483d683-7ad3-4a80-8f42-c611a88753d7_biasIsr_1123729_12.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b483d683-7ad3-4a80-8f42-c611a88753d7_biasIsr_1123729_12.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App cpBiasCombine, waiting on task 644, task 645, task 646, task 647, task 648, task 649, task 650, task 651, task 652, task 653, task 654, task 655, task 656, task 657, task 658, task 659, task 660, task 661, task 662, task 663, task 664, task 665 parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 666 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 0fedabe41cb3bd0ed2e8dcbe1352206f parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6fe9de92-19d3-40c1-80e3-bb27d8c4889a_biasIsr_1123728_5.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/6fe9de92-19d3-40c1-80e3-bb27d8c4889a_biasIsr_1123728_5.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash aef32e0748f537e69771bd122dc0573a parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/93dd4803-79c1-41c3-949e-45de6a81e0fa_biasIsr_1123835_5.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/93dd4803-79c1-41c3-949e-45de6a81e0fa_biasIsr_1123835_5.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 367a3f0eb4c3fb5e7892b80a2a75f2ab parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/28fe7e51-c7cc-4e48-92f7-c59c63dc468c_biasIsr_1123734_5.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/28fe7e51-c7cc-4e48-92f7-c59c63dc468c_biasIsr_1123734_5.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 25ebdd2d8d6f07cae56941ad34fa9f55 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f52700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/16ca8cbb-8809-48c1-b514-6857ff16254a_biasIsr_1123738_5.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/16ca8cbb-8809-48c1-b514-6857ff16254a_biasIsr_1123738_5.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 1bda101620c9b37d94f432312fd0f5f2 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f536a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e4b1cc08-7281-4abb-8bd7-91778376e4fc_biasIsr_1123733_5.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/e4b1cc08-7281-4abb-8bd7-91778376e4fc_biasIsr_1123733_5.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 9e1e315f9e8d5b87ed701ba2a494c6de parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/433646ef-8d7e-4742-936a-c624fb627f8e_biasIsr_1123730_5.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/433646ef-8d7e-4742-936a-c624fb627f8e_biasIsr_1123730_5.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash c0abe0ab54e4bdfb47e2cacf853de63c parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/9dc7ad86-0ef5-4211-be9a-5553a60df820_biasIsr_1123731_5.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/9dc7ad86-0ef5-4211-be9a-5553a60df820_biasIsr_1123731_5.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 163ba6affc6e8b24f6130419af9c67dc parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f537e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/22302181-8b1a-45d8-b9f1-71c1394d904e_biasIsr_1123830_5.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/22302181-8b1a-45d8-b9f1-71c1394d904e_biasIsr_1123830_5.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 3c2a4791b644a272868dfd1cf17e2a32 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f5f02e08-a13e-4a71-bf0a-ff3be49556f2_biasIsr_1123836_5.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f5f02e08-a13e-4a71-bf0a-ff3be49556f2_biasIsr_1123836_5.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash f8f23d50e670309a66ff4a3850e7000d parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f539c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/46e35200-6478-4b99-b55d-ddc2d7435c01_biasIsr_1123832_5.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/46e35200-6478-4b99-b55d-ddc2d7435c01_biasIsr_1123832_5.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 052aedcada299c84817498c61cee1b44 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/16a5bf75-7fa8-47cc-a6a3-66a86fd8928a_biasIsr_1123735_5.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/16a5bf75-7fa8-47cc-a6a3-66a86fd8928a_biasIsr_1123735_5.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash b862fcdac23f793c46c9c989711c30c0 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/57890d1d-cd6f-4553-abe3-9b03e4060076_biasIsr_1123837_5.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/57890d1d-cd6f-4553-abe3-9b03e4060076_biasIsr_1123837_5.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 9f16379520b34df15c37373ab8c31e4b parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/9ff55b5f-0814-437a-ab76-8040b0f4b7fe_biasIsr_1123737_5.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/9ff55b5f-0814-437a-ab76-8040b0f4b7fe_biasIsr_1123737_5.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 9cf1928ae79c645de1f0a02eee837aa3 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d7446ecb-c63e-4c3a-92ea-62693602658f_biasIsr_1123838_5.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d7446ecb-c63e-4c3a-92ea-62693602658f_biasIsr_1123838_5.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 5c764ca8049c288672c4d08e968aa323 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/828f71d5-18ef-4971-a46e-673015f6a90e_biasIsr_1123829_5.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/828f71d5-18ef-4971-a46e-673015f6a90e_biasIsr_1123829_5.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 4ecaf4e82f435518ab87b116fe767e3d parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/57e51ed7-1912-4f2b-ab6c-b1ecea04574d_biasIsr_1123729_5.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/57e51ed7-1912-4f2b-ab6c-b1ecea04574d_biasIsr_1123729_5.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 6b6c9f340adf81daf2b626e2f43eafaa parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/8acdd1ee-96a4-4cca-88ec-2932a6ea7719_biasIsr_1123839_5.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/8acdd1ee-96a4-4cca-88ec-2932a6ea7719_biasIsr_1123839_5.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 4f85fb9b4825feda8236197b76e3deb7 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/627a65a0-1f8f-4a88-a7c9-2e31d591769b_biasIsr_1123831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/627a65a0-1f8f-4a88-a7c9-2e31d591769b_biasIsr_1123831_5.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash a5d4fc2141eca03eefcbc6edb1253912 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3572ad15-d1fc-4d1a-be52-171013b875c9_biasIsr_1123834_5.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3572ad15-d1fc-4d1a-be52-171013b875c9_biasIsr_1123834_5.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 5589b9c19b6996a47bc76a4de13fb749 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9f53f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d3949be3-daca-4aef-b3b8-b063ebe331cc_biasIsr_1123736_5.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d3949be3-daca-4aef-b3b8-b063ebe331cc_biasIsr_1123736_5.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 2b2c0b46288db724cb3a6622a45a1d6a parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/95a2c585-ce32-4e49-be8d-2cf40d488e3a_biasIsr_1123732_5.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/95a2c585-ce32-4e49-be8d-2cf40d488e3a_biasIsr_1123732_5.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 248a9906b7355d5c8a065422526f11a9 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/00c86ded-fb3b-473c-9034-e30265273097_biasIsr_1123833_5.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/00c86ded-fb3b-473c-9034-e30265273097_biasIsr_1123833_5.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App cpBiasCombine, waiting on task 667, task 668, task 669, task 670, task 671, task 672, task 673, task 674, task 675, task 676, task 677, task 678, task 679, task 680, task 681, task 682, task 683, task 684, task 685, task 686, task 687, task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 6214c3d359fe85dd6683e96c5a67f570 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3c476bae-8da1-4db7-8cb8-c7e6e775e363_biasIsr_1123830_6.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3c476bae-8da1-4db7-8cb8-c7e6e775e363_biasIsr_1123830_6.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash a3192c760279b87a205a373da6e89d78 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/90dc2f8c-ce19-49ec-a31b-7891f4016afe_biasIsr_1123831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/90dc2f8c-ce19-49ec-a31b-7891f4016afe_biasIsr_1123831_6.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash fa79368005137b664a6a81ded5b6b31f parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/54181cb5-b501-49cd-89fd-d7cb650b0cc8_biasIsr_1123832_6.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/54181cb5-b501-49cd-89fd-d7cb650b0cc8_biasIsr_1123832_6.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 49fa9103079ea5a90a6db5244ce3ac95 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5e18584f-53bc-44fc-9c29-74738217ec9f_biasIsr_1123829_6.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5e18584f-53bc-44fc-9c29-74738217ec9f_biasIsr_1123829_6.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash eb1263718104b64a2bef30d429d43ad2 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f33ba74e-6b8d-4ecd-91ea-fc20294e4a16_biasIsr_1123735_6.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f33ba74e-6b8d-4ecd-91ea-fc20294e4a16_biasIsr_1123735_6.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 5e9980914364de94fa5e1b8d06ba903b parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a559706f-20f2-49c6-94fa-4cfce87f2671_biasIsr_1123729_6.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a559706f-20f2-49c6-94fa-4cfce87f2671_biasIsr_1123729_6.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 949693ef2449201abb1c5cb1cc26dc12 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1a7f2565-0c43-49a4-9a91-3036fb8a55d9_biasIsr_1123728_6.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1a7f2565-0c43-49a4-9a91-3036fb8a55d9_biasIsr_1123728_6.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 47645855819244eb935b9034ef3de2c4 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e0db6e23-5eb6-4486-bcb2-e06fcb6334c7_biasIsr_1123836_6.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/e0db6e23-5eb6-4486-bcb2-e06fcb6334c7_biasIsr_1123836_6.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 7e1af0717f4a1779390bf5c28f53ab16 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/03eaf011-57d0-45a2-ac8c-c2264c377be7_biasIsr_1123731_6.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/03eaf011-57d0-45a2-ac8c-c2264c377be7_biasIsr_1123731_6.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash dfc53b2bfce51efe4fbc089c7c9e480e parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/22318f0e-f8d6-49ac-bd84-45266220d365_biasIsr_1123738_6.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/22318f0e-f8d6-49ac-bd84-45266220d365_biasIsr_1123738_6.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 0607c08cb07e839a8cbd03f0369db6a8 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/e6b921fc-d0e4-4dd4-9228-b6b354d00ed2_biasIsr_1123839_6.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/e6b921fc-d0e4-4dd4-9228-b6b354d00ed2_biasIsr_1123839_6.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 0d52afe76da237d9cf6b1f12375d699e parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6959fee4-4d6f-45f9-979d-4ba262602f3a_biasIsr_1123734_6.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6959fee4-4d6f-45f9-979d-4ba262602f3a_biasIsr_1123734_6.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 42b2939357e206805a361103627f9b9a parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0979742d-760e-479a-b6d6-2cb4ecf10f6d_biasIsr_1123737_6.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0979742d-760e-479a-b6d6-2cb4ecf10f6d_biasIsr_1123737_6.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash b1c43beb4253c5e9810a2a9446cc391c parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa52fa2b-ab6e-4fb0-b3f8-285619b0c7e2_biasIsr_1123838_6.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa52fa2b-ab6e-4fb0-b3f8-285619b0c7e2_biasIsr_1123838_6.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 722c98afef4a414306f78bec34d6555a parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9ad70797-c50f-4108-b843-6c64aaae7837_biasIsr_1123834_6.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9ad70797-c50f-4108-b843-6c64aaae7837_biasIsr_1123834_6.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash cf10dea60bbab5742c956b3d7c5d7eba parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d992febf-c7b2-4318-b966-f72ffa06d46b_biasIsr_1123835_6.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d992febf-c7b2-4318-b966-f72ffa06d46b_biasIsr_1123835_6.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash df8a860a20f2fbad1920101e590c55a6 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/694cd6c4-4de6-420e-bed8-9355e14ae126_biasIsr_1123736_6.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/694cd6c4-4de6-420e-bed8-9355e14ae126_biasIsr_1123736_6.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 748a3cd0391515c9a6fd1c8cc4646597 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/f91735e6-3f72-4795-8360-514b57520133_biasIsr_1123833_6.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/f91735e6-3f72-4795-8360-514b57520133_biasIsr_1123833_6.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 1fbdaca4bc004edf91f4a09763137e67 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/1e753d97-7388-4b71-9378-207ced77174e_biasIsr_1123732_6.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/1e753d97-7388-4b71-9378-207ced77174e_biasIsr_1123732_6.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 47d3c52c5b1d6e1d4bb00b32671b8929 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/3b289981-eb54-4bf4-8ad2-ddf6441328b7_biasIsr_1123733_6.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/3b289981-eb54-4bf4-8ad2-ddf6441328b7_biasIsr_1123733_6.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 72739fcee606597df2a25dc2a4f9f632 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ecbb8d3d-02d0-48bf-a1f7-9222fcf5d851_biasIsr_1123730_6.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ecbb8d3d-02d0-48bf-a1f7-9222fcf5d851_biasIsr_1123730_6.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 4e60c9a200b676af7909fe886c39ad2a parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/bef5b288-d893-4884-b29e-1936503875d1_biasIsr_1123837_6.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/bef5b288-d893-4884-b29e-1936503875d1_biasIsr_1123837_6.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App cpBiasCombine, waiting on task 690, task 691, task 692, task 693, task 694, task 695, task 696, task 697, task 698, task 699, task 700, task 701, task 702, task 703, task 704, task 705, task 706, task 707, task 708, task 709, task 710, task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 744be810db383619f48db7b717128573 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9a0bc494-019c-4600-a38a-f863d133263b_biasIsr_1123736_56.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9a0bc494-019c-4600-a38a-f863d133263b_biasIsr_1123736_56.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash a30d973f351f604e07f6d162d59a5d04 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/555c42d1-7411-46aa-b5ae-44e01d8d8275_biasIsr_1123732_56.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/555c42d1-7411-46aa-b5ae-44e01d8d8275_biasIsr_1123732_56.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 3c3e47bdfe04126155f4d0f7d0112915 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/589940ee-cad1-4112-af84-99277e80033c_biasIsr_1123829_56.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/589940ee-cad1-4112-af84-99277e80033c_biasIsr_1123829_56.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 0a3ccca7d30d25bd00f7bdf6b7ef6340 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9da9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/88c62631-ce16-4513-a296-0c664f5b493f_biasIsr_1123735_56.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/88c62631-ce16-4513-a296-0c664f5b493f_biasIsr_1123735_56.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 40c3a5b4f0fcaf62f5ad0af65af1e17e parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/13a5c31e-85ff-4120-971f-cbe22363c6e7_biasIsr_1123731_56.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/13a5c31e-85ff-4120-971f-cbe22363c6e7_biasIsr_1123731_56.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 6d214e98c4b6e46219109e969265e867 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a2557e24-3d3b-49d4-8a24-a08c95987871_biasIsr_1123738_56.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a2557e24-3d3b-49d4-8a24-a08c95987871_biasIsr_1123738_56.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash adcec61966094351ecac32d9aaf50fd6 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9daaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/122a8684-8672-4573-9632-01430de723b4_biasIsr_1123730_56.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/122a8684-8672-4573-9632-01430de723b4_biasIsr_1123730_56.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash c3f424e45c301f46790f790623ebbbb5 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b9550b4e-f5f8-4839-9914-6ffafec5e09d_biasIsr_1123834_56.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b9550b4e-f5f8-4839-9914-6ffafec5e09d_biasIsr_1123834_56.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 0633fda20b1cb1e13b357e9067ca0488 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c88ae9c7-cb1a-42a7-a1b5-29fb8aefe73c_biasIsr_1123832_56.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c88ae9c7-cb1a-42a7-a1b5-29fb8aefe73c_biasIsr_1123832_56.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 2699ea8ee48fbefd7cc3e1f912efb2e6 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/dd0762e5-4095-4635-965f-365ae05a46f6_biasIsr_1123733_56.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/dd0762e5-4095-4635-965f-365ae05a46f6_biasIsr_1123733_56.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash aba6a19703c6bf4cda1f2eafb4b6e085 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/d1ca16ec-d88c-4e7c-a0cc-5dd477c96595_biasIsr_1123837_56.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/d1ca16ec-d88c-4e7c-a0cc-5dd477c96595_biasIsr_1123837_56.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash fe7d88950b15965d612f13dbfb9dd9bd parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/39b520c1-850b-498e-b262-b9077a8ee5fe_biasIsr_1123835_56.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/39b520c1-850b-498e-b262-b9077a8ee5fe_biasIsr_1123835_56.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash dd5b98d57b309a685c44816a037999f3 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/cc8a1f0f-58c2-47be-bd39-c5d8a5255088_biasIsr_1123729_56.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/cc8a1f0f-58c2-47be-bd39-c5d8a5255088_biasIsr_1123729_56.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 8bd48632aebe6f51ef9ef49e857524c2 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/36a5bc98-1667-4785-8d36-b7fb17ddc11d_biasIsr_1123836_56.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/36a5bc98-1667-4785-8d36-b7fb17ddc11d_biasIsr_1123836_56.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash e3fe3ecf0cdd344958ebfad7d7f03df0 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/0011d6d8-b913-4393-98ac-e83000307090_biasIsr_1123839_56.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/0011d6d8-b913-4393-98ac-e83000307090_biasIsr_1123839_56.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash afc7eb44b542485d584604b4fde68906 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/8ceb1d25-a0ec-4697-bce7-c6993fd588d9_biasIsr_1123831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/8ceb1d25-a0ec-4697-bce7-c6993fd588d9_biasIsr_1123831_56.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 6cf5a54aaf2a3aa7d5c913a73ec7ac74 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/a5ec517e-3bba-4134-a222-bfff2fc906da_biasIsr_1123737_56.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/a5ec517e-3bba-4134-a222-bfff2fc906da_biasIsr_1123737_56.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 5f4f58fde2bf96b0c65cd289f928d74e parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e5b9470b-de69-454d-ad80-dba3216a6bf0_biasIsr_1123728_56.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e5b9470b-de69-454d-ad80-dba3216a6bf0_biasIsr_1123728_56.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash f9e16a706a15175310db52233e44649b parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/a9750694-a44d-4b59-8267-c3ff4500ae5a_biasIsr_1123833_56.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/a9750694-a44d-4b59-8267-c3ff4500ae5a_biasIsr_1123833_56.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash f1bd482c9292051f30fe62c5fb836401 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/28ded274-0b69-4a0a-a42d-dfbf2da14c61_biasIsr_1123838_56.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/28ded274-0b69-4a0a-a42d-dfbf2da14c61_biasIsr_1123838_56.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 333ebd1e76e03eecdc84b89e56b62f89 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/07eb1d4a-2a25-403a-80eb-28f75ede6d53_biasIsr_1123734_56.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/07eb1d4a-2a25-403a-80eb-28f75ede6d53_biasIsr_1123734_56.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash a503ad5656b50a3b8997c87dc08c8f34 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9dab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/da356df0-432d-460d-a444-def4a6e1ce81_biasIsr_1123830_56.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/da356df0-432d-460d-a444-def4a6e1ce81_biasIsr_1123830_56.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App cpBiasCombine, waiting on task 713, task 714, task 715, task 716, task 717, task 718, task 719, task 720, task 721, task 722, task 723, task 724, task 725, task 726, task 727, task 728, task 729, task 730, task 731, task 732, task 733, task 734 parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 735 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 18acb8b73c6805657a627e9ee8ae5629 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ad8112ac-9bb0-4cf0-8fc2-21ad7d662997_biasIsr_1123835_57.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ad8112ac-9bb0-4cf0-8fc2-21ad7d662997_biasIsr_1123835_57.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash b9cba51f6797aca35d38434c23f73cc2 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b6f914d9-1f38-4258-9cce-7845c9841d50_biasIsr_1123731_57.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b6f914d9-1f38-4258-9cce-7845c9841d50_biasIsr_1123731_57.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash d0033871afb43541b1a1fe5d5b8413d8 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 706 parsl.dataflow.strategy DEBUG: Executor multi has 706 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 176 more blocks parsl.executors.status_handling INFO: Scaling out by 176 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-38.1709793862.6227589.submit parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-39.1709793862.6232345.submit parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-40.1709793862.6236753.submit parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-41.1709793862.6241145.submit parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-42.1709793862.6245418.submit parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-43.1709793862.6249723.submit parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-44.1709793862.6253977.submit parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-45.1709793862.6258237.submit parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-46.1709793862.6262531.submit parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-47.1709793862.6266797.submit parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-48.1709793862.6271174.submit parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-49.1709793862.6275413.submit parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-50.1709793862.6279693.submit parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-51.1709793862.6283972.submit parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-52.1709793862.6288228.submit parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-53.1709793862.6292431.submit parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-54.1709793862.6296659.submit parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-55.1709793862.6300876.submit parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-56.1709793862.630509.submit parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-57.1709793862.6309335.submit parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-58.1709793862.6313572.submit parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-59.1709793862.6317716.submit parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-60.1709793862.6322029.submit parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-61.1709793862.6326246.submit parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-62.1709793862.6330526.submit parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-63.1709793862.6334703.submit parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-64.1709793862.6338928.submit parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-65.1709793862.6343076.submit parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-66.1709793862.6347256.submit parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-67.1709793862.6351547.submit parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-68.1709793862.6355731.submit parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-69.1709793862.636001.submit parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-70.1709793862.6364202.submit parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-71.1709793862.6368427.submit parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-72.1709793862.6372628.submit parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-73.1709793862.6376834.submit parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/16ec86ee-dc30-4e32-b52c-f5e577e0a799_biasIsr_1123838_57.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/16ec86ee-dc30-4e32-b52c-f5e577e0a799_biasIsr_1123838_57.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 3b504d87a48ae27e859ca5a06448cfd6 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-74.1709793862.661738.submit parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-75.1709793862.6622.submit parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-76.1709793862.6626592.submit parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-77.1709793862.6631265.submit parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-78.1709793862.6635523.submit parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-79.1709793862.66398.submit parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-80.1709793862.6644006.submit parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-81.1709793862.664826.submit parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-82.1709793862.665246.submit parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-83.1709793862.665661.submit parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-84.1709793862.6660976.submit parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-85.1709793862.6665218.submit parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-86.1709793862.666949.submit parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-87.1709793862.6673813.submit parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-88.1709793862.6678116.submit parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-89.1709793862.668235.submit parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-90.1709793862.6686523.submit parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-91.1709793862.6690748.submit parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-92.1709793862.6694925.submit parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-93.1709793862.6809967.submit parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-94.1709793862.6814544.submit parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/dbbb7fae-b317-451b-9279-32d605ba6e64_biasIsr_1123734_57.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/dbbb7fae-b317-451b-9279-32d605ba6e64_biasIsr_1123734_57.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 7f7724c7a0bd22e2ce28bf62a6de38f9 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-95.1709793862.6818976.submit parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-96.1709793862.6981559.submit parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-97.1709793862.6985977.submit parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-98.1709793862.6990335.submit parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-99.1709793862.6994603.submit parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-100.1709793862.69989.submit parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-101.1709793862.7003155.submit parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-102.1709793862.7007406.submit parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-103.1709793862.7011697.submit parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-104.1709793862.7015903.submit parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 709 parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-105.1709793862.7020214.submit parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a2b231e7-9fc1-43c0-88cf-0e5c1ba17fcc_biasIsr_1123836_57.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a2b231e7-9fc1-43c0-88cf-0e5c1ba17fcc_biasIsr_1123836_57.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash c33e042be9357452b13fc29dc66649aa parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0043481d-5313-46cf-a408-5d7a6f91a2e3_biasIsr_1123830_57.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0043481d-5313-46cf-a408-5d7a6f91a2e3_biasIsr_1123830_57.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 6154ae699da567357758a70af4018ddc parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-106.1709793862.722764.submit parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-107.1709793862.7322712.submit parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-108.1709793862.73272.submit parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-109.1709793862.7331574.submit parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-110.1709793862.7335865.submit parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-111.1709793862.7340124.submit parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-112.1709793862.734431.submit parsl.executors.status_handling INFO: Allocated block ID 113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-113.1709793862.7348506.submit parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-114.1709793862.7352688.submit parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-115.1709793862.7356849.submit parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-116.1709793862.736107.submit parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-117.1709793862.7365212.submit parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-118.1709793862.7369459.submit parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-119.1709793862.7373612.submit parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-120.1709793862.7377794.submit parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-121.1709793862.7382054.submit parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-122.1709793862.738623.submit parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-123.1709793862.7390435.submit parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-124.1709793862.7394583.submit parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-125.1709793862.7398767.submit parsl.executors.status_handling INFO: Allocated block ID 126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-126.1709793862.7403002.submit parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-127.1709793862.7407138.submit parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-128.1709793862.7411392.submit parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-129.1709793862.7415557.submit parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-130.1709793862.7419746.submit parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-131.1709793862.742393.submit parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-132.1709793862.752942.submit parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-133.1709793862.7534556.submit parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-134.1709793862.7538874.submit parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-135.1709793862.7543108.submit parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-136.1709793862.7547312.submit parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-137.1709793862.7551548.submit parsl.executors.status_handling INFO: Allocated block ID 138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-138.1709793862.7555702.submit parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-139.1709793862.75599.submit parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-140.1709793862.756415.submit parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-141.1709793862.75684.submit parsl.executors.status_handling INFO: Allocated block ID 142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-142.1709793862.757259.submit parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-143.1709793862.7576754.submit parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-144.1709793862.7581017.submit parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-145.1709793862.758525.submit parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-146.1709793862.7589505.submit parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-147.1709793862.759367.submit parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-148.1709793862.7597806.submit parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-149.1709793862.7602022.submit parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-150.1709793862.7606175.submit parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-151.1709793862.7610378.submit parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-152.1709793862.7614608.submit parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-153.1709793862.7618802.submit parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-154.1709793862.7622936.submit parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-155.1709793862.7627053.submit parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-156.1709793862.763136.submit parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-157.1709793862.7635584.submit parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-158.1709793862.7741451.submit parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-159.1709793862.7745936.submit parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-160.1709793862.775022.submit parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-161.1709793862.7754447.submit parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-162.1709793862.7758777.submit parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-163.1709793862.7762985.submit parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-164.1709793862.7767136.submit parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-165.1709793862.777137.submit parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-166.1709793862.777552.submit parsl.executors.status_handling INFO: Allocated block ID 167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-167.1709793862.7779698.submit parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-168.1709793862.7783856.submit parsl.executors.status_handling INFO: Allocated block ID 169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-169.1709793862.7788057.submit parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-170.1709793862.7792296.submit parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-171.1709793862.7796447.submit parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-172.1709793862.78007.submit parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-173.1709793862.7804863.submit parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-174.1709793862.7809534.submit parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-175.1709793862.7813938.submit parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-176.1709793862.8048775.submit parsl.executors.status_handling INFO: Allocated block ID 177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-177.1709793862.8053617.submit parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-178.1709793862.8057911.submit parsl.executors.status_handling INFO: Allocated block ID 179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-179.1709793862.8062217.submit parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-180.1709793862.8066423.submit parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-181.1709793862.8070753.submit parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-182.1709793862.807497.submit parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-183.1709793862.807921.submit parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-184.1709793862.8083365.submit parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-185.1709793862.8087506.submit parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-186.1709793862.8091774.submit parsl.executors.status_handling INFO: Allocated block ID 187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-187.1709793862.8095946.submit parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-188.1709793862.8100162.submit parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-189.1709793862.810434.submit parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-190.1709793862.8108528.submit parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-191.1709793862.8112712.submit parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-192.1709793862.8116858.submit parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-193.1709793862.8121145.submit parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-194.1709793862.8125372.submit parsl.executors.status_handling INFO: Allocated block ID 195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-195.1709793862.81296.submit parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-196.1709793862.813375.submit parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-197.1709793862.8137875.submit parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-198.1709793862.814253.submit parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-199.1709793862.8146694.submit parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-200.1709793862.8150935.submit parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-201.1709793862.8155103.submit parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-202.1709793862.8159273.submit parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-203.1709793862.8163447.submit parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-204.1709793862.8167608.submit parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-205.1709793862.8171873.submit parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-206.1709793862.817608.submit parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-207.1709793862.81803.submit parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-208.1709793862.8184469.submit parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-209.1709793862.8188705.submit parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-210.1709793862.8192914.submit parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-211.1709793862.8197012.submit parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-212.1709793862.8201182.submit parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-06 22:44:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-213.1709793862.8205268.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/65ebaec2-b0b2-425a-841f-54d53c345e71_biasIsr_1123732_57.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/65ebaec2-b0b2-425a-841f-54d53c345e71_biasIsr_1123732_57.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash cf88c1e8d2b31f88ae72f2daa64a391a parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/684bb60a-e13d-4bac-980f-96abb1689019_biasIsr_1123729_57.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/684bb60a-e13d-4bac-980f-96abb1689019_biasIsr_1123729_57.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 777a30957007242b0ed5c3a5c5f018ed parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/337205dd-10ab-4f89-bc16-18c6fd8acde9_biasIsr_1123730_57.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/337205dd-10ab-4f89-bc16-18c6fd8acde9_biasIsr_1123730_57.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 3de9ddfb273db48ea1e94a940a8208e6 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1db4dd48-7c02-4bdb-adfa-9a880932b12d_biasIsr_1123738_57.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1db4dd48-7c02-4bdb-adfa-9a880932b12d_biasIsr_1123738_57.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 2ed371289f51ade865a82dd226b19903 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/07d0007a-cbc6-484c-a90d-b33440419cbf_biasIsr_1123833_57.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/07d0007a-cbc6-484c-a90d-b33440419cbf_biasIsr_1123833_57.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 6a2c1e82781ae6df904fc4a501229285 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/4039402c-f192-4db0-a0d4-8467d8413717_biasIsr_1123839_57.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/4039402c-f192-4db0-a0d4-8467d8413717_biasIsr_1123839_57.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash b955299fc0e1b4cb91f1b6fc386a6348 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5506d885-22bc-4739-8540-e68082e0e3a4_biasIsr_1123728_57.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5506d885-22bc-4739-8540-e68082e0e3a4_biasIsr_1123728_57.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash abae601d55f265776412684c052f40c3 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/968af65b-7a3a-48d4-a42c-3a7f3317401b_biasIsr_1123737_57.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/968af65b-7a3a-48d4-a42c-3a7f3317401b_biasIsr_1123737_57.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 03305e5dd22f3dc49e0cb20e8d0072ba parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/41e9ed63-aa37-4828-8f92-166a5998d9be_biasIsr_1123831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/41e9ed63-aa37-4828-8f92-166a5998d9be_biasIsr_1123831_57.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 785f16fc463623cef9ffd9727116783e parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/093e4eab-7017-42db-a8bf-3e058d2a816d_biasIsr_1123832_57.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/093e4eab-7017-42db-a8bf-3e058d2a816d_biasIsr_1123832_57.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 8a76fadb56b08448a52905369e804a81 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/50522671-2b72-4783-a250-d85658af152f_biasIsr_1123733_57.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/50522671-2b72-4783-a250-d85658af152f_biasIsr_1123733_57.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 4dbdd0af7f329d0824c58bec1441aa1a parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/7802173c-6e51-4d9d-8d54-49f1da97e9d2_biasIsr_1123837_57.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/7802173c-6e51-4d9d-8d54-49f1da97e9d2_biasIsr_1123837_57.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash f96d605264004d902d3279cd3a350641 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/8e2383eb-c43c-4175-8761-0363c7d3b56a_biasIsr_1123735_57.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/8e2383eb-c43c-4175-8761-0363c7d3b56a_biasIsr_1123735_57.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash b475dc6ffdae237af4f49c88d379516c parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ca58a779-7147-4b75-ae24-2eec6dfbdb58_biasIsr_1123834_57.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ca58a779-7147-4b75-ae24-2eec6dfbdb58_biasIsr_1123834_57.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash d843b59525a315d8ff8b139251174fa4 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b2b3f04a-0d9e-4622-8ae6-e9a515c07b19_biasIsr_1123829_57.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b2b3f04a-0d9e-4622-8ae6-e9a515c07b19_biasIsr_1123829_57.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 596594ad34e584691f56b980d7d68299 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5654db3f-beb6-40b7-aa57-6bad9598514a_biasIsr_1123736_57.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/5654db3f-beb6-40b7-aa57-6bad9598514a_biasIsr_1123736_57.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App cpBiasCombine, waiting on task 736, task 737, task 738, task 739, task 740, task 741, task 742, task 743, task 744, task 745, task 746, task 747, task 748, task 749, task 750, task 751, task 752, task 753, task 754, task 755, task 756, task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 6fc42f3b8f55be42a3c6cb18bc12a207 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c7e61fbe-7e57-4bb0-8f1f-663063768175_biasIsr_1123835_33.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c7e61fbe-7e57-4bb0-8f1f-663063768175_biasIsr_1123835_33.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 0d2058d001b7bb7e12da8c36d88dfebd parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a19662ff-4d15-4191-add7-72d310da2083_biasIsr_1123729_33.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a19662ff-4d15-4191-add7-72d310da2083_biasIsr_1123729_33.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash f3f2a80c4c5d9bc7040c275fa7d07bf2 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a24500ee-286e-4ac1-b8b8-6235bcaaf889_biasIsr_1123735_33.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a24500ee-286e-4ac1-b8b8-6235bcaaf889_biasIsr_1123735_33.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash d38fb40a27ada11b55325daefb869608 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ec21e698-026b-4c75-ae05-eeb72677878f_biasIsr_1123836_33.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ec21e698-026b-4c75-ae05-eeb72677878f_biasIsr_1123836_33.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash f1a38f369186d1587bc0f4a363989765 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/8f3fb958-686c-4fe7-bbec-83bb65a30181_biasIsr_1123832_33.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/8f3fb958-686c-4fe7-bbec-83bb65a30181_biasIsr_1123832_33.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 1fb5219c655b3c5b706176d8f9750026 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/decb68b2-6dd4-46b6-bf42-35bb3258f601_biasIsr_1123831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/decb68b2-6dd4-46b6-bf42-35bb3258f601_biasIsr_1123831_33.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash edfcf5421db61b7c243ee4029fff1ca0 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/389e8632-c12e-490c-8a92-6dbb60f3237b_biasIsr_1123839_33.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/389e8632-c12e-490c-8a92-6dbb60f3237b_biasIsr_1123839_33.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 64e144bbf891d62c35147e86dbdeb3ac parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/32685739-7a7e-4eb3-bf6b-136aa4eff2b6_biasIsr_1123838_33.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/32685739-7a7e-4eb3-bf6b-136aa4eff2b6_biasIsr_1123838_33.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash e24e5703945902c4f31d36ea74c7cba3 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/54d1ae10-3dd3-4a7e-8509-e9c9f2edb229_biasIsr_1123728_33.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/54d1ae10-3dd3-4a7e-8509-e9c9f2edb229_biasIsr_1123728_33.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 08f5b2d143d2766540e9f61ec3fdaaeb parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a3d5d439-1fd5-4f3b-b164-03c52c0537cd_biasIsr_1123738_33.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a3d5d439-1fd5-4f3b-b164-03c52c0537cd_biasIsr_1123738_33.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 98c0634362af329ad2c9cfc28cdc89ac parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/29db44d0-545f-400b-a63b-2390b042d4fb_biasIsr_1123736_33.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/29db44d0-545f-400b-a63b-2390b042d4fb_biasIsr_1123736_33.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash c07168558d5b233341ae2afa5b1cd257 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/da00909d-a433-445b-82cd-c6d0844d0858_biasIsr_1123829_33.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/da00909d-a433-445b-82cd-c6d0844d0858_biasIsr_1123829_33.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash e13a4b5b4ca8b6fb41157e68bdc3ef43 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/da6faef8-e24d-4193-93d4-752778585d7f_biasIsr_1123830_33.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/da6faef8-e24d-4193-93d4-752778585d7f_biasIsr_1123830_33.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 215aefe2c6e3811cdc474c94f83ef9ef parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/78231d05-8543-4a26-9848-20455d9b8134_biasIsr_1123837_33.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/78231d05-8543-4a26-9848-20455d9b8134_biasIsr_1123837_33.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 452296f9359320d13ab0381e85e35048 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/e40c9f75-63b1-4ff6-bf85-e8a79f85b4f5_biasIsr_1123731_33.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/e40c9f75-63b1-4ff6-bf85-e8a79f85b4f5_biasIsr_1123731_33.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash a23a9dc2baf2fa0f5641cd24c31b3347 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/794879ee-04d5-4e76-99db-06d8c898a77a_biasIsr_1123733_33.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/794879ee-04d5-4e76-99db-06d8c898a77a_biasIsr_1123733_33.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 4597742ad8a35eb584fc6267a864245b parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/60dfb237-db8b-44ed-b4cc-f6db490ca63b_biasIsr_1123732_33.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/60dfb237-db8b-44ed-b4cc-f6db490ca63b_biasIsr_1123732_33.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 755f78a85f5efe8832139f265dbbc829 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b928542c-5834-4332-89d1-2764136de0d7_biasIsr_1123730_33.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b928542c-5834-4332-89d1-2764136de0d7_biasIsr_1123730_33.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash a0adb683c9068abdf8b02efc1bb633d4 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cfb50757-2de4-413f-b0ad-0068333d5d26_biasIsr_1123833_33.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cfb50757-2de4-413f-b0ad-0068333d5d26_biasIsr_1123833_33.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash d8b360c7c405200ee74e6454b98e5364 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d09a954d-13f5-49d4-b18c-f21743f5292d_biasIsr_1123734_33.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d09a954d-13f5-49d4-b18c-f21743f5292d_biasIsr_1123734_33.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 60b78bf4abb506cfc1285aa14d532ddb parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/de3f860f-392b-4e04-ac08-c54a0021f52f_biasIsr_1123737_33.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/de3f860f-392b-4e04-ac08-c54a0021f52f_biasIsr_1123737_33.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 3fb1a9a56f60f7cdb9f0aa2f2110a37a parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9df3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6a0143df-06a5-4e07-acfa-e75f54d6c330_biasIsr_1123834_33.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6a0143df-06a5-4e07-acfa-e75f54d6c330_biasIsr_1123834_33.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App cpBiasCombine, waiting on task 759, task 760, task 761, task 762, task 763, task 764, task 765, task 766, task 767, task 768, task 769, task 770, task 771, task 772, task 773, task 774, task 775, task 776, task 777, task 778, task 779, task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash bd60906b793b695137aa7976459fcc41 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d68d2a5e-e9a1-40bf-8dff-29ccd4b0d223_biasIsr_1123737_55.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d68d2a5e-e9a1-40bf-8dff-29ccd4b0d223_biasIsr_1123737_55.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 834efcdf09f67cd3837af7190de2086b parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/8f430624-3281-41bb-a86b-1e980d11dd00_biasIsr_1123730_55.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/8f430624-3281-41bb-a86b-1e980d11dd00_biasIsr_1123730_55.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash c26e24130b0f4d3a9dee43c022b66a69 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/99663ca9-44e3-40d6-b0b4-2353464f41bb_biasIsr_1123835_55.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/99663ca9-44e3-40d6-b0b4-2353464f41bb_biasIsr_1123835_55.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash f6681f67f44a36e76eea41f63eccd7b7 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/40436b15-7095-4cbc-8c4b-13084467381e_biasIsr_1123732_55.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/40436b15-7095-4cbc-8c4b-13084467381e_biasIsr_1123732_55.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 3758f69e545a21084d81a5abf40d088e parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/08e16be1-39a3-4806-a311-62193fdd555a_biasIsr_1123834_55.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/08e16be1-39a3-4806-a311-62193fdd555a_biasIsr_1123834_55.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 67e760e0ec95cc3fecffc3f150c237cb parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b5485333-99c7-42c1-a9ae-9c49b70d026c_biasIsr_1123829_55.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b5485333-99c7-42c1-a9ae-9c49b70d026c_biasIsr_1123829_55.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 4b73aea2372d80c45edcff9f19010035 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/214db027-a26d-4788-a285-842d7e069dbd_biasIsr_1123833_55.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/214db027-a26d-4788-a285-842d7e069dbd_biasIsr_1123833_55.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 0b7677e42309c2098a42684e2498ba40 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/3e8f1983-ddec-4b3f-b79d-160e3923d140_biasIsr_1123734_55.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/3e8f1983-ddec-4b3f-b79d-160e3923d140_biasIsr_1123734_55.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash aaebc55f2f16e70b8aa72350c30a1808 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4c862753-f397-436c-be54-4e01777edacc_biasIsr_1123729_55.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4c862753-f397-436c-be54-4e01777edacc_biasIsr_1123729_55.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash fcdaa0ec87e265d38be29487d821b5a5 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/0d152f00-c621-4adc-9c65-ddebca49a043_biasIsr_1123735_55.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/0d152f00-c621-4adc-9c65-ddebca49a043_biasIsr_1123735_55.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 7a62ee8767e994cc94a476f1133313a3 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/16f314ee-716b-496f-9812-ba09a826089a_biasIsr_1123830_55.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/16f314ee-716b-496f-9812-ba09a826089a_biasIsr_1123830_55.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash e72bf4068f8b1891103ce27681b07ef6 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7426a4c9-54b6-43e8-864d-1b9a0a062be4_biasIsr_1123839_55.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7426a4c9-54b6-43e8-864d-1b9a0a062be4_biasIsr_1123839_55.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 4fff356bb26e6e21285049d5ead5a1c5 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cd4b8325-94d4-4328-b630-9a7df99cb03d_biasIsr_1123837_55.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cd4b8325-94d4-4328-b630-9a7df99cb03d_biasIsr_1123837_55.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash c5d9978bcf248ec49d906400a17f3b7b parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/47b980a9-279c-4c69-aa61-400829488780_biasIsr_1123738_55.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/47b980a9-279c-4c69-aa61-400829488780_biasIsr_1123738_55.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 7c06f422987f2d387c2a9aced13a9772 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/daadf835-44d4-4c21-b615-0a9ec97f9177_biasIsr_1123728_55.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/daadf835-44d4-4c21-b615-0a9ec97f9177_biasIsr_1123728_55.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash afad852f20cb0a7801baeaba09980ad9 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/439b64d7-200d-4118-953a-32106a241768_biasIsr_1123831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/439b64d7-200d-4118-953a-32106a241768_biasIsr_1123831_55.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash caec423183e4d439c4ca9dec24b3c927 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/fbbdb7e8-0f96-44fb-a272-de59e7509689_biasIsr_1123836_55.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/fbbdb7e8-0f96-44fb-a272-de59e7509689_biasIsr_1123836_55.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 7a92515ef70e2d0e3e0609d0cc0be37d parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d7d9e267-76cd-4e2b-a981-0bfd272b4da5_biasIsr_1123736_55.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/d7d9e267-76cd-4e2b-a981-0bfd272b4da5_biasIsr_1123736_55.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 5249724a1c13487707cf7d497d86b1c9 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a55e8cb1-49d5-452b-9522-7d6ef5def0e1_biasIsr_1123832_55.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a55e8cb1-49d5-452b-9522-7d6ef5def0e1_biasIsr_1123832_55.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 0d45a5f2b0bded22df386e82fbf920ae parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/3258b445-6dfc-4868-99de-26f5775bc286_biasIsr_1123838_55.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/3258b445-6dfc-4868-99de-26f5775bc286_biasIsr_1123838_55.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash b1af5f8487575f8eec30ca12e798f441 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/218cf16e-c4ff-4f73-809c-d3dff8d3d3de_biasIsr_1123733_55.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/218cf16e-c4ff-4f73-809c-d3dff8d3d3de_biasIsr_1123733_55.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash ee2291f92af761bf72b7aa049472b8ac parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57bfc595-448c-44dd-94f4-ec9667edf52b_biasIsr_1123731_55.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57bfc595-448c-44dd-94f4-ec9667edf52b_biasIsr_1123731_55.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App cpBiasCombine, waiting on task 782, task 783, task 784, task 785, task 786, task 787, task 788, task 789, task 790, task 791, task 792, task 793, task 794, task 795, task 796, task 797, task 798, task 799, task 800, task 801, task 802, task 803 parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 804 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 2d55d444c13aad8638d9038502085e6b parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e49da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d896dfaf-9ae7-405e-a7af-8d4e6cf01abf_biasIsr_1123734_28.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d896dfaf-9ae7-405e-a7af-8d4e6cf01abf_biasIsr_1123734_28.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 83c4964f005aebbb42087a82de89ff54 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e49d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/83a3cebc-375e-4e20-8c85-199f930ceb8c_biasIsr_1123831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/83a3cebc-375e-4e20-8c85-199f930ceb8c_biasIsr_1123831_28.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 63143f2db9b3eaf37f1931c89623d5e5 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e49ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/42a6d99c-dc0c-462e-b134-b3e47fd801a0_biasIsr_1123738_28.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/42a6d99c-dc0c-462e-b134-b3e47fd801a0_biasIsr_1123738_28.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 717340374b44c806a54468744e083165 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e48fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/30938d4e-97af-4d47-8852-25a5fbae112f_biasIsr_1123732_28.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/30938d4e-97af-4d47-8852-25a5fbae112f_biasIsr_1123732_28.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 386faed56cc713a828512583af83033a parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e49f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/01a76016-d12a-41c3-98c6-5a97c55a63a1_biasIsr_1123837_28.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/01a76016-d12a-41c3-98c6-5a97c55a63a1_biasIsr_1123837_28.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 8febefd470855dcfc24bfba359cca53d parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/516c51e0-1cea-4819-86d3-9016e17c1747_biasIsr_1123737_28.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/516c51e0-1cea-4819-86d3-9016e17c1747_biasIsr_1123737_28.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash bc952388e70547e4411b9cc6695cbfb2 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/ec49c0f5-4ca7-4bcf-9e16-0e716dbba7bc_biasIsr_1123735_28.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/ec49c0f5-4ca7-4bcf-9e16-0e716dbba7bc_biasIsr_1123735_28.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash c6b030626596e16335767ea8fadc847d parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57afe350-6e69-405f-814f-1e2962581c29_biasIsr_1123731_28.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/57afe350-6e69-405f-814f-1e2962581c29_biasIsr_1123731_28.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 4d1cb2280fc162d0c8aae95b88391663 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a2711eaf-ec25-49ce-bf21-38227f521cdc_biasIsr_1123836_28.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a2711eaf-ec25-49ce-bf21-38227f521cdc_biasIsr_1123836_28.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 326a8ad81143bc888a2990b8851b0d3f parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f57cb219-04e4-4633-8c1f-b50bb8f37c84_biasIsr_1123838_28.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/f57cb219-04e4-4633-8c1f-b50bb8f37c84_biasIsr_1123838_28.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash bda65b922dd8bb693841940b13f56c39 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/56c69532-c656-4984-a0e5-7e73456ce036_biasIsr_1123729_28.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/56c69532-c656-4984-a0e5-7e73456ce036_biasIsr_1123729_28.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 9f4879e131e99675a8171af235515b8e parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7734d364-76ad-4666-877f-948be82f58d4_biasIsr_1123830_28.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7734d364-76ad-4666-877f-948be82f58d4_biasIsr_1123830_28.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash a28a589e3bc96ee680c9c01845dd8d9e parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/851e5b3d-c4aa-4272-bf33-60215bf9276e_biasIsr_1123833_28.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/851e5b3d-c4aa-4272-bf33-60215bf9276e_biasIsr_1123833_28.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash f75ea0b69e840cecf6b3d26915b31139 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8d3e776e-3e30-4f59-86ca-9034197f3563_biasIsr_1123728_28.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8d3e776e-3e30-4f59-86ca-9034197f3563_biasIsr_1123728_28.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 88937bdfb52795830250ad6695ab3355 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/31034102-0b0a-4d5d-8896-20176843f357_biasIsr_1123835_28.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/31034102-0b0a-4d5d-8896-20176843f357_biasIsr_1123835_28.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 6857595529a77f855214bc4de3e6b7e7 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/523aefbc-a9a7-4bf5-80b8-a332ab013c73_biasIsr_1123733_28.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/523aefbc-a9a7-4bf5-80b8-a332ab013c73_biasIsr_1123733_28.stderr parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 528672d3889b3ee7cb1c74cb4277adff parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/69443933-4eb4-4acb-bdc6-020f02c6af2f_biasIsr_1123829_28.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/69443933-4eb4-4acb-bdc6-020f02c6af2f_biasIsr_1123829_28.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 00f2fc510557ee2dc9f303fa1b3d190d parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/f2e7a9a7-8579-43d5-8c0f-9189d146c598_biasIsr_1123736_28.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/f2e7a9a7-8579-43d5-8c0f-9189d146c598_biasIsr_1123736_28.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash dcf931fcafc55009b786bfbb23685db3 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d168e159-9201-4e5e-9cda-3ec90aa6b6ac_biasIsr_1123832_28.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d168e159-9201-4e5e-9cda-3ec90aa6b6ac_biasIsr_1123832_28.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 4b0d7b7f8254c1eb537734bcd1144b81 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9aae1e35-277e-4d28-8b28-c17d5696efef_biasIsr_1123834_28.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9aae1e35-277e-4d28-8b28-c17d5696efef_biasIsr_1123834_28.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 15dcc3ae1daca85417f7ef01fe60a3e8 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/85506b4a-c8e2-4395-b015-a6fceabb92c0_biasIsr_1123839_28.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/85506b4a-c8e2-4395-b015-a6fceabb92c0_biasIsr_1123839_28.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 50c26aa16b64ed94349491933d09d1ac parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/dc24579b-6d39-4467-9a72-b0d07c90c1f8_biasIsr_1123730_28.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/dc24579b-6d39-4467-9a72-b0d07c90c1f8_biasIsr_1123730_28.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App cpBiasCombine, waiting on task 805, task 806, task 807, task 808, task 809, task 810, task 811, task 812, task 813, task 814, task 815, task 816, task 817, task 818, task 819, task 820, task 821, task 822, task 823, task 824, task 825, task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash d98d9f481bb074269294da66b3b8f0ad parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/ff1c63e9-ba30-4712-abb4-941103192b20_biasIsr_1123833_22.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/ff1c63e9-ba30-4712-abb4-941103192b20_biasIsr_1123833_22.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 5f443d7781dd37444e3ed522a6e40742 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/75b5c87f-4664-4a5c-ac23-67b7f7854d23_biasIsr_1123732_22.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/75b5c87f-4664-4a5c-ac23-67b7f7854d23_biasIsr_1123732_22.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 15d0d536eb514ba2cb382773d7704259 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/daa86d2f-fb0b-4545-ac03-48686aad6c4a_biasIsr_1123735_22.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/daa86d2f-fb0b-4545-ac03-48686aad6c4a_biasIsr_1123735_22.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 8894f640a7e1714b893a717d53c37c49 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/f27e713b-5360-459f-9613-d5d82d970472_biasIsr_1123834_22.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/f27e713b-5360-459f-9613-d5d82d970472_biasIsr_1123834_22.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 829164b3e23a796f74a0b39087527cf5 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/3a811c39-cb53-4dfc-9f93-2e6c4506dfcc_biasIsr_1123728_22.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/3a811c39-cb53-4dfc-9f93-2e6c4506dfcc_biasIsr_1123728_22.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash c7f90b315c3e1a15753271ff116746fd parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/911541b2-1bb2-49bb-9d1b-79353da4d60e_biasIsr_1123832_22.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/911541b2-1bb2-49bb-9d1b-79353da4d60e_biasIsr_1123832_22.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash 36179db17b0a21e3281e134280945748 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/c442f979-000b-4fc2-a292-58d6a3c08e90_biasIsr_1123830_22.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/c442f979-000b-4fc2-a292-58d6a3c08e90_biasIsr_1123830_22.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash a7da346d130e73b25e9580e9f1f07c47 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/d178ef46-d1d2-431d-b305-32e370d68c5a_biasIsr_1123831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/d178ef46-d1d2-431d-b305-32e370d68c5a_biasIsr_1123831_22.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 5cb187c381843666b3ec7b8144f6b863 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/cee255c9-86cd-474b-83b9-f8a64d1c674d_biasIsr_1123736_22.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/cee255c9-86cd-474b-83b9-f8a64d1c674d_biasIsr_1123736_22.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash e26247d6f6b10a3f25f1c865e9bd2a14 parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2ae54986-3b4d-4c0d-a098-81a66c338639_biasIsr_1123836_22.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2ae54986-3b4d-4c0d-a098-81a66c338639_biasIsr_1123836_22.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 456d5a29e26944f6b21f9ba16d9ea675 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9e4bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/6f8daeb6-51f4-4e43-a867-d05fcd2ecd02_biasIsr_1123731_22.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/6f8daeb6-51f4-4e43-a867-d05fcd2ecd02_biasIsr_1123731_22.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash d2eead6836263231192af02ec279725c parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c980e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1828d382-a5cd-46ff-b600-77c6daefd533_biasIsr_1123829_22.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/1828d382-a5cd-46ff-b600-77c6daefd533_biasIsr_1123829_22.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash fa7fff57411b4b54055a5817393e14d4 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8f10c103-59ce-47e3-86ff-5f93da0458fb_biasIsr_1123734_22.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/8f10c103-59ce-47e3-86ff-5f93da0458fb_biasIsr_1123734_22.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash a380033cf285710505478cb6e3dece68 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/76cf5069-2ac5-4245-a44c-3097c650dc5e_biasIsr_1123838_22.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/76cf5069-2ac5-4245-a44c-3097c650dc5e_biasIsr_1123838_22.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash a124a6b9e90da72485a2563dfcc7ef63 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e5430ce3-ffa0-47b6-b8da-3d2813385bb8_biasIsr_1123835_22.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e5430ce3-ffa0-47b6-b8da-3d2813385bb8_biasIsr_1123835_22.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 6e9a43c1c11d492448073d7cc4b00ad8 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d37617e8-c20f-4fb6-abf2-fc4f161db9f6_biasIsr_1123729_22.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d37617e8-c20f-4fb6-abf2-fc4f161db9f6_biasIsr_1123729_22.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash afc1e8012b397516bb0ce6693457ba15 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c3da451e-55c5-46ff-a88b-ea63a2b6cb84_biasIsr_1123837_22.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c3da451e-55c5-46ff-a88b-ea63a2b6cb84_biasIsr_1123837_22.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash e43af7a41806af0ddca98570f183771c parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6cef2505-f140-4b7f-bfcd-30869781c1b9_biasIsr_1123737_22.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/6cef2505-f140-4b7f-bfcd-30869781c1b9_biasIsr_1123737_22.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 17a6d8c0391f53a01e520300ca8ff52c parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d0473d6c-68c3-465e-8d1a-7dba8fd87a04_biasIsr_1123839_22.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d0473d6c-68c3-465e-8d1a-7dba8fd87a04_biasIsr_1123839_22.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 5c828ddfd9717a8e0e23b73deb6ce024 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/495d5574-64a9-448c-9ec2-da3392656ca4_biasIsr_1123733_22.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/495d5574-64a9-448c-9ec2-da3392656ca4_biasIsr_1123733_22.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 53804d61a8742c67b89bef8859d2ec5a parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c985e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/774c139d-813b-4fd5-a4ab-b00cee6c966a_biasIsr_1123738_22.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/774c139d-813b-4fd5-a4ab-b00cee6c966a_biasIsr_1123738_22.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash afdb6f9527e91ac0f1695fcfd02a0007 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/3b187751-89a4-47ea-bded-03a5577aab5b_biasIsr_1123730_22.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/3b187751-89a4-47ea-bded-03a5577aab5b_biasIsr_1123730_22.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App cpBiasCombine, waiting on task 828, task 829, task 830, task 831, task 832, task 833, task 834, task 835, task 836, task 837, task 838, task 839, task 840, task 841, task 842, task 843, task 844, task 845, task 846, task 847, task 848, task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash f521e392e5517f5eb763eb3b1e460b90 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/77163114-de34-4b32-8ae1-ccbef594258d_biasIsr_1123831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/77163114-de34-4b32-8ae1-ccbef594258d_biasIsr_1123831_38.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 6f67b29dccbe8ac033ef6dd4c3db441c parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/789e8c17-1638-41f4-9309-2032783bfd28_biasIsr_1123730_38.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/789e8c17-1638-41f4-9309-2032783bfd28_biasIsr_1123730_38.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 178efad0556a8fb9b1dfa5cedec89ab7 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0f0ed47d-dc12-4a3c-8741-23f2ec134e9f_biasIsr_1123830_38.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0f0ed47d-dc12-4a3c-8741-23f2ec134e9f_biasIsr_1123830_38.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash d07fa1e8938261663eda82bed17479b5 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/75dfde99-d2e8-4aab-8118-8b3bffe40a9a_biasIsr_1123834_38.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/75dfde99-d2e8-4aab-8118-8b3bffe40a9a_biasIsr_1123834_38.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 95cf2a9842d1df929ebd1f8d6d716b10 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c98900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/14c9cf84-2399-4cf0-a9d3-3ae9967ded51_biasIsr_1123837_38.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/14c9cf84-2399-4cf0-a9d3-3ae9967ded51_biasIsr_1123837_38.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 3ec7759d1cd2f6a362003adec48cb0f8 parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c998a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6c659c1a-404b-44b3-b8e1-62200a26d0d7_biasIsr_1123832_38.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/6c659c1a-404b-44b3-b8e1-62200a26d0d7_biasIsr_1123832_38.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 6f07e64f815842118914c4b17391adc2 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/a38b8e77-f168-4587-9348-eef388f9f695_biasIsr_1123839_38.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/a38b8e77-f168-4587-9348-eef388f9f695_biasIsr_1123839_38.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash a096f80f96caca5d5696d41462fb0b31 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c999e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/973ce630-f156-4ef9-a091-ec2cf9077866_biasIsr_1123736_38.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/973ce630-f156-4ef9-a091-ec2cf9077866_biasIsr_1123736_38.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 0d1eadd0177f7a9bfbe8702462a328e3 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/cf2ad0a2-7c4a-468b-8807-a95d2a5f8cf6_biasIsr_1123731_38.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/cf2ad0a2-7c4a-468b-8807-a95d2a5f8cf6_biasIsr_1123731_38.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 70a936cfbbab594b783985df08e3dd89 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/ce3892cb-ee6f-468b-9e8a-49afcfcd5456_biasIsr_1123735_38.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/ce3892cb-ee6f-468b-9e8a-49afcfcd5456_biasIsr_1123735_38.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash f223ff3f6c7408a289cfd4cdaef236b2 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/bc84ef40-6c72-4b4d-b8ca-c89465b6cee4_biasIsr_1123733_38.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/bc84ef40-6c72-4b4d-b8ca-c89465b6cee4_biasIsr_1123733_38.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash e80bf62b3fb5a20672a1c5939d9521b4 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/2c246eca-3ee2-4846-bcc4-af929d1149a8_biasIsr_1123838_38.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/2c246eca-3ee2-4846-bcc4-af929d1149a8_biasIsr_1123838_38.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash fda9efc4b25fc8761d078253a100af54 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/efdb82cf-56d1-46dd-9283-14f32ebc7b2d_biasIsr_1123829_38.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/efdb82cf-56d1-46dd-9283-14f32ebc7b2d_biasIsr_1123829_38.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash c632b15244d8b253060c2da844e55adc parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3cbd576b-80b0-4dcc-b2a3-988e8b5d7a97_biasIsr_1123835_38.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3cbd576b-80b0-4dcc-b2a3-988e8b5d7a97_biasIsr_1123835_38.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 250d29ff08931dcd929c7eae71c113a8 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d3ca14fe-6a3c-442e-9747-67284dcf8d73_biasIsr_1123729_38.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d3ca14fe-6a3c-442e-9747-67284dcf8d73_biasIsr_1123729_38.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b4248a6c392147bbae30f3ec4d6d43cc parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/15735bd4-1981-4409-840c-64a4de0b9d9b_biasIsr_1123737_38.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/15735bd4-1981-4409-840c-64a4de0b9d9b_biasIsr_1123737_38.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash dedcaa49e4cb4e4f057ebf85606bf5dc parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c99f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8b9bfe84-96dd-419c-a7f2-668a52ebb148_biasIsr_1123738_38.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8b9bfe84-96dd-419c-a7f2-668a52ebb148_biasIsr_1123738_38.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 269357146b9e6e350a883485fe21acae parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/0e8973a5-30fc-4cfe-8504-87c2a2b53fd9_biasIsr_1123734_38.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/0e8973a5-30fc-4cfe-8504-87c2a2b53fd9_biasIsr_1123734_38.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash ae05aff8b30920120e02187398e92bb5 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1c33a895-1736-4a69-8485-c0bfe5127da2_biasIsr_1123833_38.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1c33a895-1736-4a69-8485-c0bfe5127da2_biasIsr_1123833_38.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 378df6a42dca33f476cb54a90d8b64f8 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8752b1e0-04d6-4873-b97f-3d419ff1a192_biasIsr_1123836_38.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8752b1e0-04d6-4873-b97f-3d419ff1a192_biasIsr_1123836_38.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash d6d68fd6107d44c0dc80a06fdc8f633d parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/d62cd53c-dcc5-4146-a3f5-c650bc4153c3_biasIsr_1123732_38.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/d62cd53c-dcc5-4146-a3f5-c650bc4153c3_biasIsr_1123732_38.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 758c255b80ebee22b5f144bb8525313a parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/b7a89a19-7e94-4e9b-bde2-15f12dee5861_biasIsr_1123728_38.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/b7a89a19-7e94-4e9b-bde2-15f12dee5861_biasIsr_1123728_38.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App cpBiasCombine, waiting on task 851, task 852, task 853, task 854, task 855, task 856, task 857, task 858, task 859, task 860, task 861, task 862, task 863, task 864, task 865, task 866, task 867, task 868, task 869, task 870, task 871, task 872 parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 873 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 606661f326a954a94f5c01e64abb2f2d parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0754c193-e54d-4663-a0b7-9fb5956bf13f_biasIsr_1123830_45.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/0754c193-e54d-4663-a0b7-9fb5956bf13f_biasIsr_1123830_45.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 345b773e535aabfc096d5bb1523c1a79 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7c4a28f7-073d-4b8b-9cd9-87f024e85a5e_biasIsr_1123839_45.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/7c4a28f7-073d-4b8b-9cd9-87f024e85a5e_biasIsr_1123839_45.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 0bf13929404a386b75727e4242c78dc0 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2e6b3bbd-9563-4815-9b36-e2d577ce0c67_biasIsr_1123836_45.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2e6b3bbd-9563-4815-9b36-e2d577ce0c67_biasIsr_1123836_45.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 0b5e7294d3752c35faf1692f41a8ea2a parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/0700dd62-9b89-4e70-9960-09fdf6fa73a9_biasIsr_1123835_45.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/0700dd62-9b89-4e70-9960-09fdf6fa73a9_biasIsr_1123835_45.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 19a62ead43ead63d8aa1765d9bd8000c parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7351afaa-0ef3-45f5-a45e-2557fa9e161d_biasIsr_1123832_45.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7351afaa-0ef3-45f5-a45e-2557fa9e161d_biasIsr_1123832_45.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash b3fc0e0d304b3770b2cdfeaaf69bd298 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/74dd3afc-aec0-4709-b6b8-5b70960a7938_biasIsr_1123728_45.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/74dd3afc-aec0-4709-b6b8-5b70960a7938_biasIsr_1123728_45.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 69efb36a40603a9a18e4f210730e9f82 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1a3f7fc7-b0ea-4560-880f-fd60ecbb43bd_biasIsr_1123738_45.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/1a3f7fc7-b0ea-4560-880f-fd60ecbb43bd_biasIsr_1123738_45.stderr parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 1b24faacab1ca42bbacf48176ac1325c parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/eb2141c7-7c20-419a-831d-f442f8ffcaeb_biasIsr_1123834_45.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/eb2141c7-7c20-419a-831d-f442f8ffcaeb_biasIsr_1123834_45.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash d8719a6f5696a998a787b9a135da1e76 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/f41b3bc1-b898-4695-b413-5637aa4a9d41_biasIsr_1123829_45.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/f41b3bc1-b898-4695-b413-5637aa4a9d41_biasIsr_1123829_45.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 4165329b76ffaf975cbff39de2671ddd parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/47b85eac-cd4e-4076-a611-366b95b2ca41_biasIsr_1123734_45.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/47b85eac-cd4e-4076-a611-366b95b2ca41_biasIsr_1123734_45.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 03c87a321f6752eb8bbcb3261b27685f parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fbdbca09-1ced-4998-9d08-12292f2022f5_biasIsr_1123732_45.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/fbdbca09-1ced-4998-9d08-12292f2022f5_biasIsr_1123732_45.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash bbc4a4834159456a166d6c6f252f00c9 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1862c677-145d-4293-a0c6-8504a6183f97_biasIsr_1123729_45.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1862c677-145d-4293-a0c6-8504a6183f97_biasIsr_1123729_45.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash c9b18d06dec5f494c42c267a993b2da4 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/32e415ad-9e18-41cc-945d-7c3a182985ed_biasIsr_1123731_45.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/32e415ad-9e18-41cc-945d-7c3a182985ed_biasIsr_1123731_45.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 353b072b646166f17202093436777260 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c11aa1a0-a222-4bd5-a2d9-ed95d46c6750_biasIsr_1123831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/c11aa1a0-a222-4bd5-a2d9-ed95d46c6750_biasIsr_1123831_45.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash af4a3f754720e5bcf868e47e3079bbf5 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/8e4c5596-a826-4c2e-8d2d-2d0c60ee7263_biasIsr_1123833_45.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/8e4c5596-a826-4c2e-8d2d-2d0c60ee7263_biasIsr_1123833_45.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 3278a93b1f59e124d9e69a577a18dade parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2336258e-727c-4b6d-8353-11be67a72998_biasIsr_1123837_45.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2336258e-727c-4b6d-8353-11be67a72998_biasIsr_1123837_45.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash a6e77a5106c031ef3cd6937194edf6d7 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4acfdbff-6814-4519-9c5d-71625491b791_biasIsr_1123730_45.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4acfdbff-6814-4519-9c5d-71625491b791_biasIsr_1123730_45.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash b7fbc1d5368a976e616a42f81468fb2a parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/adeccc20-7d98-4030-b1c0-82396a3829be_biasIsr_1123736_45.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/adeccc20-7d98-4030-b1c0-82396a3829be_biasIsr_1123736_45.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 7abaefab30efda2263b1fdef6ad60d97 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/1027b5fa-3fe1-43db-9ec3-c9114d4b62b6_biasIsr_1123735_45.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/1027b5fa-3fe1-43db-9ec3-c9114d4b62b6_biasIsr_1123735_45.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 188421fb851141bed3436b56cc406abb parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/605cd206-c7f0-41f5-ad26-a25c68c713c8_biasIsr_1123838_45.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/605cd206-c7f0-41f5-ad26-a25c68c713c8_biasIsr_1123838_45.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash a38cfb7dd7e594451a5bea3acd253d22 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b4653440-d93c-43e6-a56e-3f57d8c8f95e_biasIsr_1123737_45.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b4653440-d93c-43e6-a56e-3f57d8c8f95e_biasIsr_1123737_45.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash eadf20643fff8841fa6f5ee89b67f0ac parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c9bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a2e983ba-d452-4ecc-9010-3eb2818488b6_biasIsr_1123733_45.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a2e983ba-d452-4ecc-9010-3eb2818488b6_biasIsr_1123733_45.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App cpBiasCombine, waiting on task 874, task 875, task 876, task 877, task 878, task 879, task 880, task 881, task 882, task 883, task 884, task 885, task 886, task 887, task 888, task 889, task 890, task 891, task 892, task 893, task 894, task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 9b5cff9d93edfdf46e90f489b5679440 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1a7f0ca3-018a-492a-9cad-e09b341a2a1b_biasIsr_1123838_62.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1a7f0ca3-018a-492a-9cad-e09b341a2a1b_biasIsr_1123838_62.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 1d15a4baa20e286a579b877f15c40bfd parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/f02c70d0-a056-4577-b655-233975e3182d_biasIsr_1123829_62.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/f02c70d0-a056-4577-b655-233975e3182d_biasIsr_1123829_62.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash e5d17bad6553162cc23263783da6d6fb parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/87907300-8d45-4cb6-991d-95af1ee331ef_biasIsr_1123732_62.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/87907300-8d45-4cb6-991d-95af1ee331ef_biasIsr_1123732_62.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash b6557c6e5e9c6e7ce6e15ab87d73002d parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/654d7e59-09fd-4cd1-8e37-1d6e0a2b7067_biasIsr_1123833_62.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/654d7e59-09fd-4cd1-8e37-1d6e0a2b7067_biasIsr_1123833_62.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 63e896a49aab6c728e12791641e27db8 parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bc95a46f-3b33-4f7c-8bf4-b9ad11836a54_biasIsr_1123736_62.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bc95a46f-3b33-4f7c-8bf4-b9ad11836a54_biasIsr_1123736_62.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 4a1f30d3cfe9ba5a73c4e191cb2f5a90 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/00bfbf34-999e-433b-a0a1-d3ea9842d8e6_biasIsr_1123837_62.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/00bfbf34-999e-433b-a0a1-d3ea9842d8e6_biasIsr_1123837_62.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash e591e5a876d4f1eacbbbe8429d9afe7b parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/14117ada-37ea-4067-999e-15649235974b_biasIsr_1123737_62.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/14117ada-37ea-4067-999e-15649235974b_biasIsr_1123737_62.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash b353f3972b13c9eee23fd0725db7f84f parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c2d8a196-fba6-41a4-acc8-bb189105fa50_biasIsr_1123731_62.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c2d8a196-fba6-41a4-acc8-bb189105fa50_biasIsr_1123731_62.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash d3e4eb5983c762f7d5fa95a8b510bed7 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b4495292-71a3-45e9-8a4a-558dc6ba9090_biasIsr_1123836_62.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b4495292-71a3-45e9-8a4a-558dc6ba9090_biasIsr_1123836_62.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 55d12f340c7f3b6f89ab9f49c29d3a4b parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7bb770ca-16a3-4500-beff-6adffca7c17c_biasIsr_1123832_62.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7bb770ca-16a3-4500-beff-6adffca7c17c_biasIsr_1123832_62.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 5a3d89fbd406c744d2d9554819a479c1 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ac224131-4c33-47b8-b611-d9bcc9c6deef_biasIsr_1123734_62.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/ac224131-4c33-47b8-b611-d9bcc9c6deef_biasIsr_1123734_62.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 0ab769995254c1e2dfe1cdc4b8268093 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a57475f3-e8e4-487c-9c6e-7ef4c95625bb_biasIsr_1123735_62.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a57475f3-e8e4-487c-9c6e-7ef4c95625bb_biasIsr_1123735_62.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 704fdb8db876d09a5f2a6bd74ad0a943 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/bc0f717e-b3e6-4701-84bd-b763d40aa16c_biasIsr_1123835_62.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/bc0f717e-b3e6-4701-84bd-b763d40aa16c_biasIsr_1123835_62.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 1d8ff3e19b4af3289bad8b3de69062bf parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/69a1aecc-27a9-42ad-9ba1-1ae4fb32c853_biasIsr_1123834_62.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/69a1aecc-27a9-42ad-9ba1-1ae4fb32c853_biasIsr_1123834_62.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 71aeca7dff4faa80bfcfe1712a0251cc parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/19baaf7c-b970-41d2-b225-35610638f2e5_biasIsr_1123728_62.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/19baaf7c-b970-41d2-b225-35610638f2e5_biasIsr_1123728_62.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 886f371b0389b8d9a705b49f809f95d4 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7bbbd0e3-6415-4d02-872b-cc400932ad9f_biasIsr_1123729_62.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7bbbd0e3-6415-4d02-872b-cc400932ad9f_biasIsr_1123729_62.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 05208d91ae079609010919127800e2a2 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4606cc8f-e5f3-4288-b49d-3bb038c4e5fb_biasIsr_1123730_62.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/4606cc8f-e5f3-4288-b49d-3bb038c4e5fb_biasIsr_1123730_62.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 3a2821ec554b267e38298adb8046e0bd parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/553f72ae-2dcd-46c4-8360-2afc646dffdf_biasIsr_1123738_62.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/553f72ae-2dcd-46c4-8360-2afc646dffdf_biasIsr_1123738_62.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash f33a4b70aa3dcb16171c6d7acb89232c parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/922af89b-fc84-4146-beda-3a05ca5ef746_biasIsr_1123733_62.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/922af89b-fc84-4146-beda-3a05ca5ef746_biasIsr_1123733_62.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 334d1ee9566536f03f4ceacc921e23aa parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ced3ff9d-28c0-4a72-8172-90570135ab84_biasIsr_1123839_62.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ced3ff9d-28c0-4a72-8172-90570135ab84_biasIsr_1123839_62.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash e0d2dd454fd07225eab2c9502843bd81 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/4653045b-c514-4297-b9ef-a07bcdfafe7f_biasIsr_1123830_62.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/4653045b-c514-4297-b9ef-a07bcdfafe7f_biasIsr_1123830_62.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 270757d2b62900cca34a66f2f24ab06c parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/d2340d92-eee2-4c7c-a85a-ec8d72daddb2_biasIsr_1123831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/d2340d92-eee2-4c7c-a85a-ec8d72daddb2_biasIsr_1123831_62.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App cpBiasCombine, waiting on task 897, task 898, task 899, task 900, task 901, task 902, task 903, task 904, task 905, task 906, task 907, task 908, task 909, task 910, task 911, task 912, task 913, task 914, task 915, task 916, task 917, task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash c1cbd07497706f63ec21c3cf5d245d77 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/5ac21102-9fcf-4717-8c1d-fb98c1cb4a35_biasIsr_1123732_11.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/5ac21102-9fcf-4717-8c1d-fb98c1cb4a35_biasIsr_1123732_11.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 16b678a1ef8df4db5d12a3d283161ece parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e97473bb-ac34-43d3-b829-fe613c10229d_biasIsr_1123831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e97473bb-ac34-43d3-b829-fe613c10229d_biasIsr_1123831_11.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 83731bfd47c311e6b9cdf6ceb66df250 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/8d53f31f-691f-4c89-b041-6d8bb79d4039_biasIsr_1123830_11.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/8d53f31f-691f-4c89-b041-6d8bb79d4039_biasIsr_1123830_11.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 55b3647607a8cc1b68af4c551a915365 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/ac92cfa4-948d-4c85-9539-f8105e1b2cc5_biasIsr_1123832_11.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/ac92cfa4-948d-4c85-9539-f8105e1b2cc5_biasIsr_1123832_11.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 6f7585f7c20c3211063e100831e1248e parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9e924c04-f2e3-4140-b8b6-6346b78f3522_biasIsr_1123834_11.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9e924c04-f2e3-4140-b8b6-6346b78f3522_biasIsr_1123834_11.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 3b2d86e00b6ccea53d9be1be7ab6256d parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2fd28129-863c-4b02-b8bc-aeb8df9028d8_biasIsr_1123839_11.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2fd28129-863c-4b02-b8bc-aeb8df9028d8_biasIsr_1123839_11.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 17e07ecb6b17fca08383909d62f24f3e parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ebd38c95-a129-485e-9ea1-29fc7851eda0_biasIsr_1123835_11.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ebd38c95-a129-485e-9ea1-29fc7851eda0_biasIsr_1123835_11.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 4631c266a5c5d017a9f0b6f87b2fd999 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/d6d72849-8e7a-462f-9925-4056b23a0873_biasIsr_1123733_11.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/d6d72849-8e7a-462f-9925-4056b23a0873_biasIsr_1123733_11.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 92bda9e282d0c9acc46f372fea84e116 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f70a0125-1997-4df3-8deb-548014633e34_biasIsr_1123837_11.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f70a0125-1997-4df3-8deb-548014633e34_biasIsr_1123837_11.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash cb829df352168229d61a67cb279223c8 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/43bebaf8-9f40-4cca-9e8e-7cd04cf02209_biasIsr_1123734_11.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/43bebaf8-9f40-4cca-9e8e-7cd04cf02209_biasIsr_1123734_11.stderr parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 1d30f7e9263deb11aa445f848b9dd010 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/ade34916-c0f0-46d8-bec8-abb5d4a3ac87_biasIsr_1123737_11.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/ade34916-c0f0-46d8-bec8-abb5d4a3ac87_biasIsr_1123737_11.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash c8cd35288b4f42f807c60e4df67635c9 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0063c8b7-05aa-4bfb-8b2d-59420c14d6ad_biasIsr_1123731_11.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0063c8b7-05aa-4bfb-8b2d-59420c14d6ad_biasIsr_1123731_11.stderr parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 72666b2f101a78ac7bb3cd396286cc9a parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/ea24ac3c-610b-4d34-91f5-41a1abac4ce5_biasIsr_1123833_11.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/ea24ac3c-610b-4d34-91f5-41a1abac4ce5_biasIsr_1123833_11.stderr parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash ae05dbd681213fa1d5720dcb722e3c7d parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/76a3d236-dec7-4511-866d-c0e390704c29_biasIsr_1123736_11.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/76a3d236-dec7-4511-866d-c0e390704c29_biasIsr_1123736_11.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 0f3258e45b52fca3a3d0b441d1f37ba6 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/f7bac788-b3b4-4381-824d-b0371c52249e_biasIsr_1123730_11.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/f7bac788-b3b4-4381-824d-b0371c52249e_biasIsr_1123730_11.stderr parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 0e8c5c4327654a0101b3f0ed45c6d668 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c37ccc1f-3013-4bc0-86fe-2e9e8c28f358_biasIsr_1123729_11.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c37ccc1f-3013-4bc0-86fe-2e9e8c28f358_biasIsr_1123729_11.stderr parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 16bd3170b0729ee1437f34cd61269ef7 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa76be84-7c11-481a-8366-f4628b5bb2ef_biasIsr_1123838_11.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa76be84-7c11-481a-8366-f4628b5bb2ef_biasIsr_1123838_11.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash 46ebd604eb09f477b5b04e04901ae299 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/81983011-6e07-4c68-b73e-2e760d223a32_biasIsr_1123836_11.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/81983011-6e07-4c68-b73e-2e760d223a32_biasIsr_1123836_11.stderr parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 6975ca0b909ee600a38dc2f2f54c6ee7 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e0266f3e-bc80-4f55-930d-0f84e3ec772f_biasIsr_1123829_11.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e0266f3e-bc80-4f55-930d-0f84e3ec772f_biasIsr_1123829_11.stderr parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 509c70ecf72ce28e0c4dcb3407c5042c parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/30997f9c-09d8-4eb5-8c64-f881494f4b54_biasIsr_1123728_11.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/30997f9c-09d8-4eb5-8c64-f881494f4b54_biasIsr_1123728_11.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash cd6df43f8551216c7e3368913fde1d59 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f0a3483b-6212-4f7e-a0d5-7cf7ea038d47_biasIsr_1123738_11.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f0a3483b-6212-4f7e-a0d5-7cf7ea038d47_biasIsr_1123738_11.stderr parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 9e35f45b05d441b14f0463f6156a2a0f parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9ce3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/19371676-3777-4181-9228-ea4b2bffa590_biasIsr_1123735_11.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/19371676-3777-4181-9228-ea4b2bffa590_biasIsr_1123735_11.stderr parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App cpBiasCombine, waiting on task 920, task 921, task 922, task 923, task 924, task 925, task 926, task 927, task 928, task 929, task 930, task 931, task 932, task 933, task 934, task 935, task 936, task 937, task 938, task 939, task 940, task 941 parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 942 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 9b1ed9f1409857b3c9945ceec1b7161b parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/50efe5ad-61ee-4db9-94f0-7d0aa1ae0e57_biasIsr_1123839_30.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/50efe5ad-61ee-4db9-94f0-7d0aa1ae0e57_biasIsr_1123839_30.stderr parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 354717aaa843e0f09856cdf160236fd7 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/385c83d9-b441-43c9-a8df-85f7496fefda_biasIsr_1123832_30.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/385c83d9-b441-43c9-a8df-85f7496fefda_biasIsr_1123832_30.stderr parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash e1a9a2f13b1c590b35d05eb96cebb4c4 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d307c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7c3563e0-7eae-42f8-a0e0-631feb3b1c4f_biasIsr_1123733_30.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7c3563e0-7eae-42f8-a0e0-631feb3b1c4f_biasIsr_1123733_30.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash cef932ad9d423a01f8e31d5a0354e186 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b3380908-24b3-427b-9ea2-f490899c1d30_biasIsr_1123833_30.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b3380908-24b3-427b-9ea2-f490899c1d30_biasIsr_1123833_30.stderr parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash 70b9a9351a155cd3d5f3082f2aafee53 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c917e5e3-b960-4483-a338-8df4a388f207_biasIsr_1123729_30.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/c917e5e3-b960-4483-a338-8df4a388f207_biasIsr_1123729_30.stderr parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 9939fd241bcaf0b2f6d4950ce3c2a199 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d309a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/591a8570-5c8a-4d67-b681-5d65cc002426_biasIsr_1123734_30.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/591a8570-5c8a-4d67-b681-5d65cc002426_biasIsr_1123734_30.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash e10efae84e59b850c27c6ff452999dcf parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/85135743-772e-46d0-be30-98a489b14572_biasIsr_1123835_30.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/85135743-772e-46d0-be30-98a489b14572_biasIsr_1123835_30.stderr parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash b303310809755f1defe89a152a2c26a5 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1a718105-dfa0-4c38-b5a0-36c99e2d6640_biasIsr_1123836_30.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1a718105-dfa0-4c38-b5a0-36c99e2d6640_biasIsr_1123836_30.stderr parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash dc22e4f8fac079eac606c84cbb26aff2 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/4dced483-cb5e-47d0-9d6e-7f7076f9b1c1_biasIsr_1123732_30.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/4dced483-cb5e-47d0-9d6e-7f7076f9b1c1_biasIsr_1123732_30.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 03a778cc3651b978572d6fb3be324ab5 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/1b9daace-bcee-4c60-8312-4e17f3c60e5e_biasIsr_1123735_30.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/1b9daace-bcee-4c60-8312-4e17f3c60e5e_biasIsr_1123735_30.stderr parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash db36d8a95bf3e7a3d27fa1750152cc4c parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b7a3daf0-4d2d-40d6-9605-02f37227d87c_biasIsr_1123731_30.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/b7a3daf0-4d2d-40d6-9605-02f37227d87c_biasIsr_1123731_30.stderr parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 3dec19f5e61be0402b2586c4072770cc parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/59878701-fa6f-4904-8d3c-fcb5a96e95a7_biasIsr_1123728_30.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/59878701-fa6f-4904-8d3c-fcb5a96e95a7_biasIsr_1123728_30.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 19f67966b2bc5146a7cc13959fd501a3 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/9d4ed694-6375-4dac-b58f-2ab2783c77bb_biasIsr_1123738_30.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/9d4ed694-6375-4dac-b58f-2ab2783c77bb_biasIsr_1123738_30.stderr parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 8a761e94b97dbfeba6af5b0ed8d0e1a5 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/38e478a0-1573-423d-9d63-eba760726490_biasIsr_1123730_30.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/38e478a0-1573-423d-9d63-eba760726490_biasIsr_1123730_30.stderr parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 0b01f2c1d1dfc7312dac297d621ad2c6 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/80bf6a39-0098-4e13-b09f-4c0a64f112ac_biasIsr_1123829_30.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/80bf6a39-0098-4e13-b09f-4c0a64f112ac_biasIsr_1123829_30.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 1c9de709a09648b2365080ecb7d10f5d parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d30fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00d68df2-504e-4c5e-ba61-87b028f8cf93_biasIsr_1123830_30.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00d68df2-504e-4c5e-ba61-87b028f8cf93_biasIsr_1123830_30.stderr parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 4cc931aefa765de3cee631a39e17ddec parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/a30aa653-7016-4871-8349-e4c8e17c3b24_biasIsr_1123834_30.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/a30aa653-7016-4871-8349-e4c8e17c3b24_biasIsr_1123834_30.stderr parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash 6ca7dd36348c9ef9de24c4abd81b6627 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d311c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a52c743-e280-4a3d-9e93-1780c232198d_biasIsr_1123736_30.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a52c743-e280-4a3d-9e93-1780c232198d_biasIsr_1123736_30.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 795a923a470e4729962f1b22182c7a31 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ab4ae639-25aa-417b-a361-806fc0e0789e_biasIsr_1123837_30.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ab4ae639-25aa-417b-a361-806fc0e0789e_biasIsr_1123837_30.stderr parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash f21af3e290f8aa91651c6f228b97124d parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d313a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/e735b443-71f3-479c-99c1-83361316f1ff_biasIsr_1123737_30.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/e735b443-71f3-479c-99c1-83361316f1ff_biasIsr_1123737_30.stderr parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash ca8c57d4ff1d23dee4581177d962fd57 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9a3f5266-183a-48c0-b0e2-2f0629b14186_biasIsr_1123838_30.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9a3f5266-183a-48c0-b0e2-2f0629b14186_biasIsr_1123838_30.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash dd72a740baa342acfae63b6aa2c86982 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f5026465-b05c-4548-9af2-4eee1ccc1751_biasIsr_1123831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f5026465-b05c-4548-9af2-4eee1ccc1751_biasIsr_1123831_30.stderr parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App cpBiasCombine, waiting on task 943, task 944, task 945, task 946, task 947, task 948, task 949, task 950, task 951, task 952, task 953, task 954, task 955, task 956, task 957, task 958, task 959, task 960, task 961, task 962, task 963, task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 24ce56d9fad9fff570410384cd1551f2 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/e6733336-5f5d-408a-9e1b-92e1e9f2a54c_biasIsr_1123838_9.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/e6733336-5f5d-408a-9e1b-92e1e9f2a54c_biasIsr_1123838_9.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash d17079e5309c364003c3de203834fd2c parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d322a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5f6b6fe6-4416-4246-a718-606ff0c2116e_biasIsr_1123829_9.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/5f6b6fe6-4416-4246-a718-606ff0c2116e_biasIsr_1123829_9.stderr parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash efd30c05d53f017fc7d9802699d7c5c2 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/2eb2aa73-b09b-47db-a0b8-9d2a107d94b7_biasIsr_1123729_9.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/2eb2aa73-b09b-47db-a0b8-9d2a107d94b7_biasIsr_1123729_9.stderr parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 85150b4fb12f485c3a2c0b4c0793834e parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/13d3b236-a644-4a9e-af7a-dc314e1ed99c_biasIsr_1123730_9.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/13d3b236-a644-4a9e-af7a-dc314e1ed99c_biasIsr_1123730_9.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 10a5b90c1bbeb1af0815776cc3c06f82 parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d31620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/30a0edfb-b3f9-486b-81a9-a896ce403a0f_biasIsr_1123832_9.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/30a0edfb-b3f9-486b-81a9-a896ce403a0f_biasIsr_1123832_9.stderr parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 118275225f40b47ce6312241175b3fdf parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d325c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c171d41a-517f-4032-b77d-38a1ae430fb0_biasIsr_1123837_9.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c171d41a-517f-4032-b77d-38a1ae430fb0_biasIsr_1123837_9.stderr parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 13a43c43af5a38f0e04aca2b4a28dc51 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/39d1c60e-d0ea-4d40-b7a0-09d5d84e6c4f_biasIsr_1123738_9.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/39d1c60e-d0ea-4d40-b7a0-09d5d84e6c4f_biasIsr_1123738_9.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 256aa7d84c92111521e3b6dc5dfbf2ac parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c2281e0f-3356-4f05-b2f1-cc09a67003f0_biasIsr_1123732_9.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c2281e0f-3356-4f05-b2f1-cc09a67003f0_biasIsr_1123732_9.stderr parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash e59c60795b58f70beb898c81bd6af5c6 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/040c848f-f79c-4bed-aebe-dbbc2155aa6d_biasIsr_1123833_9.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/040c848f-f79c-4bed-aebe-dbbc2155aa6d_biasIsr_1123833_9.stderr parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 3392a0e72c0fe1fb136434b33a294318 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d327a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/11e2dd82-c79a-45d8-8c87-fa2ae07c1252_biasIsr_1123839_9.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/11e2dd82-c79a-45d8-8c87-fa2ae07c1252_biasIsr_1123839_9.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 90af585b35f50907d59f2488870db76f parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d328e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/265325ce-e15e-49f7-923b-09ef963a321a_biasIsr_1123830_9.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/265325ce-e15e-49f7-923b-09ef963a321a_biasIsr_1123830_9.stderr parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 2072be39382578272182be9c6498bdd1 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/1a64cd0b-a88d-46d1-95ef-fd377df9c97c_biasIsr_1123736_9.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/1a64cd0b-a88d-46d1-95ef-fd377df9c97c_biasIsr_1123736_9.stderr parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 34e91df30a5ba24fbd8f51b49c6ff557 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5b590c2b-17a6-4d62-8dee-e3ce196b365f_biasIsr_1123836_9.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5b590c2b-17a6-4d62-8dee-e3ce196b365f_biasIsr_1123836_9.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 2184579b77cfd233eac98a2887eceaf3 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/2fee0e8d-83ab-4b91-8cfb-d91064998dc7_biasIsr_1123735_9.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/2fee0e8d-83ab-4b91-8cfb-d91064998dc7_biasIsr_1123735_9.stderr parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 853ad0ef449d4f5822cc893cb87150d7 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e5deadf7-9c4e-4e81-b550-471bb26d3db6_biasIsr_1123835_9.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e5deadf7-9c4e-4e81-b550-471bb26d3db6_biasIsr_1123835_9.stderr parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 7e8e258a1f91a144f130500b2cd5f4d9 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/286418aa-a72a-48cd-bbd3-c336193e77eb_biasIsr_1123831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/286418aa-a72a-48cd-bbd3-c336193e77eb_biasIsr_1123831_9.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash ac5902f599044ac7694abaf840214f76 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8d139956-6937-4138-90e4-8759630318ac_biasIsr_1123733_9.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8d139956-6937-4138-90e4-8759630318ac_biasIsr_1123733_9.stderr parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 79fd9f5d959670a722b922f4c4c1e2dc parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cad826d8-c6a8-44d7-82c9-7f2b62c66741_biasIsr_1123737_9.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cad826d8-c6a8-44d7-82c9-7f2b62c66741_biasIsr_1123737_9.stderr parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 09ff281dc2c4952782797cbd7a789da3 parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9c639f22-41be-44b4-ac23-6d52b3951924_biasIsr_1123834_9.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/9c639f22-41be-44b4-ac23-6d52b3951924_biasIsr_1123834_9.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 7b2094eabc142d91f39260a27299e0ba parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/32f905b7-19ed-45f4-b9eb-11859f361f3a_biasIsr_1123728_9.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/32f905b7-19ed-45f4-b9eb-11859f361f3a_biasIsr_1123728_9.stderr parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 262d3ac59ac48d041df85c6115ca5c46 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/45f08f26-756b-4aba-adf0-5e36001fad69_biasIsr_1123734_9.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/45f08f26-756b-4aba-adf0-5e36001fad69_biasIsr_1123734_9.stderr parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 0f47a5016aa86a8579f56718df396164 parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d32fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/1a215b71-3466-4438-b167-92f49b21dd96_biasIsr_1123731_9.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/1a215b71-3466-4438-b167-92f49b21dd96_biasIsr_1123731_9.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App cpBiasCombine, waiting on task 966, task 967, task 968, task 969, task 970, task 971, task 972, task 973, task 974, task 975, task 976, task 977, task 978, task 979, task 980, task 981, task 982, task 983, task 984, task 985, task 986, task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 3640e5221f352ba62d0358523f332ba8 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d33f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/9e11a803-2c37-499a-9cdf-f81a5d6bffa9_biasIsr_1123829_42.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/9e11a803-2c37-499a-9cdf-f81a5d6bffa9_biasIsr_1123829_42.stderr parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 7735f9a4b7801b513a4dc13eb7d876da parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d33240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/268a1df8-31b6-483f-8019-ad7339525b50_biasIsr_1123728_42.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/268a1df8-31b6-483f-8019-ad7339525b50_biasIsr_1123728_42.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash f8ad9aa5fe8fd529ab23e11bcdac3825 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/83e658bd-6d82-4406-9fd4-5af02ccec1d5_biasIsr_1123732_42.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/83e658bd-6d82-4406-9fd4-5af02ccec1d5_biasIsr_1123732_42.stderr parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 74d63782cdb7d54ec3a9675e7daaab68 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/787f02d7-0c56-425a-8b61-32ef0a6d63ab_biasIsr_1123734_42.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/787f02d7-0c56-425a-8b61-32ef0a6d63ab_biasIsr_1123734_42.stderr parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 4f6901337d23dac9cab50b6a4a0e528e parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/40963e5a-98fa-4e11-8abf-bd7a1e686362_biasIsr_1123830_42.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/40963e5a-98fa-4e11-8abf-bd7a1e686362_biasIsr_1123830_42.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 550e64824f140c5fa1a698230d2840d1 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aaa7cb43-35b7-48f2-93d1-32ee546d3523_biasIsr_1123738_42.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aaa7cb43-35b7-48f2-93d1-32ee546d3523_biasIsr_1123738_42.stderr parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 3333393ceadaaf077ebb36ac2797bad4 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d702c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f46ea321-f428-4b0a-ba27-8e7f9aa8ea0d_biasIsr_1123737_42.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f46ea321-f428-4b0a-ba27-8e7f9aa8ea0d_biasIsr_1123737_42.stderr parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash d49d9e6e2b29cc9ba31bd07c52106492 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3d057b81-2603-4319-993a-b6d41a2b7703_biasIsr_1123835_42.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/3d057b81-2603-4319-993a-b6d41a2b7703_biasIsr_1123835_42.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash b94f48dec6fcb7b6b015916a665e01d5 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d704a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/de7b3806-78f0-4db6-8453-4e09794f8738_biasIsr_1123839_42.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/de7b3806-78f0-4db6-8453-4e09794f8738_biasIsr_1123839_42.stderr parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 9ffcbb5ac087a8c91167779f42eb2642 parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6fc12c2d-877c-4046-87a9-a6b5d4583bc4_biasIsr_1123735_42.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/6fc12c2d-877c-4046-87a9-a6b5d4583bc4_biasIsr_1123735_42.stderr parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 8c73d39056afafd0954db2fbef7ae845 parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/63bf04f8-bb11-4b2e-8e4f-2926bc2a445b_biasIsr_1123837_42.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/63bf04f8-bb11-4b2e-8e4f-2926bc2a445b_biasIsr_1123837_42.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash e28acde7f1f4767d0c0148bfc9f916c8 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e8bc1302-b3cb-4baa-8b6f-da33385d40d1_biasIsr_1123832_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e8bc1302-b3cb-4baa-8b6f-da33385d40d1_biasIsr_1123832_42.stderr parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 74e47ab8de1ad3e0b3a1bdbf9cc89369 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/5742a318-02b8-4ba3-bca0-d994a1a8f498_biasIsr_1123834_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/5742a318-02b8-4ba3-bca0-d994a1a8f498_biasIsr_1123834_42.stderr parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash de6e9343f1ac9e1b3aaa034d2e72353b parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d707c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/254692fc-c397-48e4-b43e-81425d5abb79_biasIsr_1123729_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/254692fc-c397-48e4-b43e-81425d5abb79_biasIsr_1123729_42.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 8bd38d0f3fa627d0e62801a6eb9af48a parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bca24658-de61-4e6e-a76e-5806f445d00f_biasIsr_1123736_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bca24658-de61-4e6e-a76e-5806f445d00f_biasIsr_1123736_42.stderr parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 1c5826ab60c89adec339527a3c54c1a5 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d709a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/86a243d9-6a69-4852-a68f-bad146679b96_biasIsr_1123731_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/86a243d9-6a69-4852-a68f-bad146679b96_biasIsr_1123731_42.stderr parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 592e2027029760b084c9fa66f4c0b756 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ff7eee6f-ec57-4658-9298-4d9a480bcdcc_biasIsr_1123838_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ff7eee6f-ec57-4658-9298-4d9a480bcdcc_biasIsr_1123838_42.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 246ccd823c7d83b81dd3836c002c6f63 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cf50e8d3-a00f-41af-ae54-bf00b7c4dc99_biasIsr_1123833_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/cf50e8d3-a00f-41af-ae54-bf00b7c4dc99_biasIsr_1123833_42.stderr parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 0dfa7db4865c66ab5d5e2ec102f01e49 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/28be66d0-ae1a-4690-a6f2-3d87d7e26ad7_biasIsr_1123733_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/28be66d0-ae1a-4690-a6f2-3d87d7e26ad7_biasIsr_1123733_42.stderr parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 24355e594ac444b5e1f12d66379266ed parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/57bfa82a-2d63-42db-aba6-35775b6e133b_biasIsr_1123730_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/57bfa82a-2d63-42db-aba6-35775b6e133b_biasIsr_1123730_42.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash abc5f4e3654798f65ad06e595ab81dd9 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/146d185e-343c-4d55-aab6-746c556e42c5_biasIsr_1123831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/146d185e-343c-4d55-aab6-746c556e42c5_biasIsr_1123831_42.stderr parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 821f0f356ea7adaf7de3b4ff75f1327d parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/834c25c5-bf29-49ab-9581-d4cf84dad64b_biasIsr_1123836_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/834c25c5-bf29-49ab-9581-d4cf84dad64b_biasIsr_1123836_42.stderr parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App cpBiasCombine, waiting on task 989, task 990, task 991, task 992, task 993, task 994, task 995, task 996, task 997, task 998, task 999, task 1000, task 1001, task 1002, task 1003, task 1004, task 1005, task 1006, task 1007, task 1008, task 1009, task 1010 parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1011 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 3f08c4eade9dc88287e115d2427219b2 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/1fe43e2b-026e-4ee7-bf83-850e48d6a824_biasIsr_1123731_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/1fe43e2b-026e-4ee7-bf83-850e48d6a824_biasIsr_1123731_21.stderr parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 4bae88073e2bc1286229061adfa89b0b parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d74c67dc-a92a-4545-b4a1-82687771a986_biasIsr_1123838_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d74c67dc-a92a-4545-b4a1-82687771a986_biasIsr_1123838_21.stderr parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 68aae0aa702497b87bbd116d1b7349c3 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/3a0b9c01-8eee-4657-a4b8-9970eb798026_biasIsr_1123837_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/3a0b9c01-8eee-4657-a4b8-9970eb798026_biasIsr_1123837_21.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash a0b5999fb53c34dfa03970c2151bfd09 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d70ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/59747161-44e3-4d75-8719-86589ec15dee_biasIsr_1123735_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/59747161-44e3-4d75-8719-86589ec15dee_biasIsr_1123735_21.stderr parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash ca27fafb16fc9a08feabc28ec3a6ca1f parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/d6fee6c5-06e9-4e8a-ac80-92a3f044cf1a_biasIsr_1123730_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/d6fee6c5-06e9-4e8a-ac80-92a3f044cf1a_biasIsr_1123730_21.stderr parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash 7919e768d8fcbc83198772854c52c175 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/710b6ddd-0250-4da4-93f4-c375da93c298_biasIsr_1123829_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/710b6ddd-0250-4da4-93f4-c375da93c298_biasIsr_1123829_21.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash dcedb2dbe169981f7a27c3057d4ca3c7 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/135fe66c-5914-4a3f-af1f-9a2add1068e8_biasIsr_1123832_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/135fe66c-5914-4a3f-af1f-9a2add1068e8_biasIsr_1123832_21.stderr parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 0753abab06f45aca2b0dbd23b24074f5 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d71f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/8b1f11db-404f-43c8-adf6-22b9800fc8ac_biasIsr_1123736_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/8b1f11db-404f-43c8-adf6-22b9800fc8ac_biasIsr_1123736_21.stderr parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash bcf8e2e2ab879124d172c5ce514acee6 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/c383133f-9add-4d64-9bf2-18248a4a18b4_biasIsr_1123728_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/c383133f-9add-4d64-9bf2-18248a4a18b4_biasIsr_1123728_21.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 7e96ecc03319b4fd654957e01edca8da parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e2d9cf0d-1d6a-49f4-ab11-46db437165bf_biasIsr_1123831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e2d9cf0d-1d6a-49f4-ab11-46db437165bf_biasIsr_1123831_21.stderr parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 442661e7472b728aca8b460b2c97b717 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/82485afe-b51d-48bc-b12d-553e27b97e25_biasIsr_1123833_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/82485afe-b51d-48bc-b12d-553e27b97e25_biasIsr_1123833_21.stderr parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash ddd55321a157214abf7861e47842be6d parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f7eeec38-f45b-4316-a4b8-0c739e4dd981_biasIsr_1123737_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/f7eeec38-f45b-4316-a4b8-0c739e4dd981_biasIsr_1123737_21.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 47fcc592c5b855239c6490fce558d396 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d722a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c7657c0d-a832-4c52-b3f4-20d74cf675d6_biasIsr_1123834_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/c7657c0d-a832-4c52-b3f4-20d74cf675d6_biasIsr_1123834_21.stderr parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 31448270078efa0dbf6d3a93bcc553ab parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/f03f6d3f-6afe-4f2b-95ee-0f6a48e9f430_biasIsr_1123734_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/f03f6d3f-6afe-4f2b-95ee-0f6a48e9f430_biasIsr_1123734_21.stderr parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash a04b293c9281d1276d93874762e663be parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/9289d089-c014-4c80-a513-4bf0c07bdaff_biasIsr_1123830_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/9289d089-c014-4c80-a513-4bf0c07bdaff_biasIsr_1123830_21.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash daacb43c729354157542fb8f5e15f31d parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/77bdd5f8-010f-4675-b3e5-d253c675e4e7_biasIsr_1123839_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/77bdd5f8-010f-4675-b3e5-d253c675e4e7_biasIsr_1123839_21.stderr parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 435cbb3ae07affdc29e5faee78f5bfbd parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b97b6d4d-0753-4145-acb7-bf1d20e2536b_biasIsr_1123729_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b97b6d4d-0753-4145-acb7-bf1d20e2536b_biasIsr_1123729_21.stderr parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 36415bbda38a9c8e3f649b5a92627735 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/c8f03f82-867e-4d5a-a2f5-d2962f2dacb4_biasIsr_1123738_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/c8f03f82-867e-4d5a-a2f5-d2962f2dacb4_biasIsr_1123738_21.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash c7906a7dbfeb03136ced5d5003baad82 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ad97cb0f-d9f7-4518-957c-fcaa4e4fe86f_biasIsr_1123836_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/ad97cb0f-d9f7-4518-957c-fcaa4e4fe86f_biasIsr_1123836_21.stderr parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 6a21df3ccfea893a7ff3b9589169fb85 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/7c6dcc84-e8f4-453a-ae4d-9cfacde39476_biasIsr_1123732_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/7c6dcc84-e8f4-453a-ae4d-9cfacde39476_biasIsr_1123732_21.stderr parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 6556786e28d925aab590be7162624645 parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/83ee57c6-cc86-45bb-b864-1a5844d31c69_biasIsr_1123733_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/83ee57c6-cc86-45bb-b864-1a5844d31c69_biasIsr_1123733_21.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 8ebad9b5af4a3183ea554a22bda26c80 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d7a12a9b-6995-4cbd-9642-714c7759b110_biasIsr_1123835_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/d7a12a9b-6995-4cbd-9642-714c7759b110_biasIsr_1123835_21.stderr parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App cpBiasCombine, waiting on task 1012, task 1013, task 1014, task 1015, task 1016, task 1017, task 1018, task 1019, task 1020, task 1021, task 1022, task 1023, task 1024, task 1025, task 1026, task 1027, task 1028, task 1029, task 1030, task 1031, task 1032, task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 1a84648914e3b32522d7ba86ab5445b3 parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/65c89c4a-a29d-42f2-b647-b9015636a00a_biasIsr_1123728_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/65c89c4a-a29d-42f2-b647-b9015636a00a_biasIsr_1123728_40.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 52a05ef2f0d94048cdb6fb5519f531f2 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b737743e-a389-4187-b9a1-28890dec8c97_biasIsr_1123730_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b737743e-a389-4187-b9a1-28890dec8c97_biasIsr_1123730_40.stderr parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 912bd797acdae8cf72d616c8d2a65b85 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6248b0ca-a1f6-4664-8ac9-e7c018777dfd_biasIsr_1123834_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6248b0ca-a1f6-4664-8ac9-e7c018777dfd_biasIsr_1123834_40.stderr parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash cf061299f7c9b3ecfced22ab32812dd4 parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d72a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/369c2753-8d6e-4cfb-966e-3ff1bd9ef302_biasIsr_1123832_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/369c2753-8d6e-4cfb-966e-3ff1bd9ef302_biasIsr_1123832_40.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash ea28b1f139b9b45690cba8974217296f parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d739c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/792c925f-8ac1-46ad-a6d6-b6a044b549e6_biasIsr_1123734_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/792c925f-8ac1-46ad-a6d6-b6a044b549e6_biasIsr_1123734_40.stderr parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 5723f5adc5010741bb30b803bcda210d parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/cfe23002-1493-4c08-8d57-dbb4f069caaf_biasIsr_1123835_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/cfe23002-1493-4c08-8d57-dbb4f069caaf_biasIsr_1123835_40.stderr parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 5d835b4ef15cc3282334cf62e93683ac parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a487c789-a07d-43f9-bf3d-4f6d889ebd6a_biasIsr_1123837_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a487c789-a07d-43f9-bf3d-4f6d889ebd6a_biasIsr_1123837_40.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 05080673447ec471dee42325dd02add8 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f5dbada8-15d9-4bf7-9a3d-7ec75aa396f9_biasIsr_1123738_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f5dbada8-15d9-4bf7-9a3d-7ec75aa396f9_biasIsr_1123738_40.stderr parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash bd191b5db900711b38234d43229226a1 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d12d0003-dc8c-41d3-a5e7-0a77e05917c5_biasIsr_1123838_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/d12d0003-dc8c-41d3-a5e7-0a77e05917c5_biasIsr_1123838_40.stderr parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 64265db7846570e6a677b1f2f91ee1a8 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2b3b71ba-fba7-4490-9cfa-033bd26f099a_biasIsr_1123737_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2b3b71ba-fba7-4490-9cfa-033bd26f099a_biasIsr_1123737_40.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash b07cc8150aa8bfc2a0cb743cf9ff8afa parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c9d9beda-d572-47c1-b9da-1583072f4684_biasIsr_1123731_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c9d9beda-d572-47c1-b9da-1583072f4684_biasIsr_1123731_40.stderr parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash c2280fbe9507e2db561e34feab438f95 parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/d3f17cd7-4295-4ba2-b4a5-aa6684790b6a_biasIsr_1123833_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/d3f17cd7-4295-4ba2-b4a5-aa6684790b6a_biasIsr_1123833_40.stderr parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash e9ccc5357cd663e70fedea3fc6aa7f1c parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8c8716ab-d01a-457b-8e4e-81dbbb8c5849_biasIsr_1123836_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8c8716ab-d01a-457b-8e4e-81dbbb8c5849_biasIsr_1123836_40.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 8e69e14ad06811f8bdd7692883675bd5 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9d73880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b28db226-4598-475b-bf53-438cba39930a_biasIsr_1123839_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/b28db226-4598-475b-bf53-438cba39930a_biasIsr_1123839_40.stderr parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 1c61eb2ff373690fe7a0f4a592b9853a parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/9d72d145-796f-4069-bec4-16b67c348cfd_biasIsr_1123831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/9d72d145-796f-4069-bec4-16b67c348cfd_biasIsr_1123831_40.stderr parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 3a8987e1eea00e74e9d64f790c712699 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/2604f6b4-9399-4a3a-908f-d2ebc7295dd5_biasIsr_1123732_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/2604f6b4-9399-4a3a-908f-d2ebc7295dd5_biasIsr_1123732_40.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash cc9ce9a36c9d447c85b017d589571dd7 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/84e8b653-5a77-4ca3-92b1-773227d9af34_biasIsr_1123829_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/84e8b653-5a77-4ca3-92b1-773227d9af34_biasIsr_1123829_40.stderr parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 5e192764d4fadc130bcc8a9c5cf80939 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/8c29d93e-fbdc-4fde-a771-f5636ea5dbcb_biasIsr_1123735_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/8c29d93e-fbdc-4fde-a771-f5636ea5dbcb_biasIsr_1123735_40.stderr parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 875d6b4e6bcac7325aa317888de95241 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/4c6fbbb5-79a2-4a85-8f89-cbdfd8ba2a4c_biasIsr_1123736_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/4c6fbbb5-79a2-4a85-8f89-cbdfd8ba2a4c_biasIsr_1123736_40.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 71012e97c3ba57aafbe2f8e72e67a9a4 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/2f5a1084-b756-4d40-a4ed-47be7dabdd08_biasIsr_1123729_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/2f5a1084-b756-4d40-a4ed-47be7dabdd08_biasIsr_1123729_40.stderr parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 55a8abac6d8a61835baaf6491834fdd3 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8c0d4498-80a3-4d8a-9001-eae55c207a79_biasIsr_1123733_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8c0d4498-80a3-4d8a-9001-eae55c207a79_biasIsr_1123733_40.stderr parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 857ee420bf87fdde99cb3cd629f89193 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3f4969c8-71b3-41c8-9579-14fbf6a21af3_biasIsr_1123830_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3f4969c8-71b3-41c8-9579-14fbf6a21af3_biasIsr_1123830_40.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App cpBiasCombine, waiting on task 1035, task 1036, task 1037, task 1038, task 1039, task 1040, task 1041, task 1042, task 1043, task 1044, task 1045, task 1046, task 1047, task 1048, task 1049, task 1050, task 1051, task 1052, task 1053, task 1054, task 1055, task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash c33ef433805398aa2d4ed9913a7cb02e parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fad39af6-3587-4a96-9af1-339abdc691a6_biasIsr_1123733_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/fad39af6-3587-4a96-9af1-339abdc691a6_biasIsr_1123733_7.stderr parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash a2f617613b4b3a4338f4a82d0bf584ba parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/be60c1dd-fd6b-462a-a058-67c6f8400969_biasIsr_1123731_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/be60c1dd-fd6b-462a-a058-67c6f8400969_biasIsr_1123731_7.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 2bcdf3cf8d9fadaad99e8373a9202ec3 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/0a651144-1704-413d-b255-dfcdb10a9582_biasIsr_1123829_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/0a651144-1704-413d-b255-dfcdb10a9582_biasIsr_1123829_7.stderr parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 00cdd29956d7acb534c3665a1ace25fc parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/4688a88a-0081-4aff-b0d1-a6ce840f579c_biasIsr_1123839_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/4688a88a-0081-4aff-b0d1-a6ce840f579c_biasIsr_1123839_7.stderr parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash e4ece9777d65f7708756a340905a02af parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/0dded89c-796d-42cd-ba84-b6dcc682b748_biasIsr_1123831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/0dded89c-796d-42cd-ba84-b6dcc682b748_biasIsr_1123831_7.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash de00c12d703bc8f787d444737eae46d2 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e0e74dfe-626f-45a3-a7a9-331e66eda982_biasIsr_1123830_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e0e74dfe-626f-45a3-a7a9-331e66eda982_biasIsr_1123830_7.stderr parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash d3bc67311f5a9424e62863161876485c parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/b8a52556-6314-4f72-a1db-42b120ed17fc_biasIsr_1123736_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/b8a52556-6314-4f72-a1db-42b120ed17fc_biasIsr_1123736_7.stderr parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash a181c2290c23ffb4a95027b77dc48af0 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a042d743-8e77-40b5-8c51-aa65cc236837_biasIsr_1123735_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a042d743-8e77-40b5-8c51-aa65cc236837_biasIsr_1123735_7.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 29c1c5752e87b5104ab22ec5bcccd975 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/e2a9d555-fa01-46e7-a65c-49f4141b5a83_biasIsr_1123730_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/e2a9d555-fa01-46e7-a65c-49f4141b5a83_biasIsr_1123730_7.stderr parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 02fdd6c85bd89323c991110d15ce159f parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/13ed90cd-bcdb-43c2-af91-290152df8338_biasIsr_1123834_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/13ed90cd-bcdb-43c2-af91-290152df8338_biasIsr_1123834_7.stderr parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 9eab81625cf155adad9d3d3cdf080b35 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d89a19aa-757f-46a1-95a3-6a90e7c62c64_biasIsr_1123832_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/d89a19aa-757f-46a1-95a3-6a90e7c62c64_biasIsr_1123832_7.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 7d3a60b0633b9f0f5f7fd0a75c59eee3 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/14f4c980-35c5-4cf6-976a-0acc6ae91ccf_biasIsr_1123734_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/14f4c980-35c5-4cf6-976a-0acc6ae91ccf_biasIsr_1123734_7.stderr parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 9b16726994443f81e325d2bfc53401bb parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ed1b0a76-67fa-453a-9612-89bbeb403d93_biasIsr_1123837_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/ed1b0a76-67fa-453a-9612-89bbeb403d93_biasIsr_1123837_7.stderr parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash c681ecb2731a848e5cad88dd27ce5598 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/0938a7f5-c89c-486f-93a4-1e4d50f82fd2_biasIsr_1123838_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/0938a7f5-c89c-486f-93a4-1e4d50f82fd2_biasIsr_1123838_7.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 097d96b35bf5f91d97c39be16e7c5327 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/a9deb608-3323-44f4-83ff-8e4b6b16417d_biasIsr_1123737_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/a9deb608-3323-44f4-83ff-8e4b6b16417d_biasIsr_1123737_7.stderr parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash a4d12c19a8e02406a1849ffe39c15771 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/635dc92e-163e-4a4c-a4eb-bdbb05cd579b_biasIsr_1123728_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/635dc92e-163e-4a4c-a4eb-bdbb05cd579b_biasIsr_1123728_7.stderr parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash bf0afef963125d87d205c2f1de54db47 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4c86383e-c8a9-4399-90a6-2bf96e36d8bd_biasIsr_1123835_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4c86383e-c8a9-4399-90a6-2bf96e36d8bd_biasIsr_1123835_7.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 88c6c6f65ff3ddec986ffea34cdd613c parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/703bb881-c006-4c43-a090-58281a90d6c6_biasIsr_1123732_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/703bb881-c006-4c43-a090-58281a90d6c6_biasIsr_1123732_7.stderr parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 8fa62ecc75d5cfe0c4933a0859ec55d0 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/3bcdbffc-6305-411b-87d2-4d640ba8d1d8_biasIsr_1123833_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/3bcdbffc-6305-411b-87d2-4d640ba8d1d8_biasIsr_1123833_7.stderr parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash c419528d9a60c28630cb33123b26b565 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bc9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/f14fca5d-4580-4d05-825f-fce1d50189e7_biasIsr_1123729_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/f14fca5d-4580-4d05-825f-fce1d50189e7_biasIsr_1123729_7.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash ee3623560cb07eb8f1da6b1c738c9770 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bca0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/cbfc343f-23c6-49dc-809d-df137a242793_biasIsr_1123738_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/cbfc343f-23c6-49dc-809d-df137a242793_biasIsr_1123738_7.stderr parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 88ecdc3bc34c695cf9ae2566d1cc446c parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bca020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/23aede1b-3a5b-4985-958b-c448110370ba_biasIsr_1123836_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/23aede1b-3a5b-4985-958b-c448110370ba_biasIsr_1123836_7.stderr parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App cpBiasCombine, waiting on task 1058, task 1059, task 1060, task 1061, task 1062, task 1063, task 1064, task 1065, task 1066, task 1067, task 1068, task 1069, task 1070, task 1071, task 1072, task 1073, task 1074, task 1075, task 1076, task 1077, task 1078, task 1079 parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1080 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash a5b0ed59745d9f46285d5b52ae7064b9 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9922e7e1-8540-4264-ab83-fd5357ac2849_biasIsr_1123838_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/9922e7e1-8540-4264-ab83-fd5357ac2849_biasIsr_1123838_43.stderr parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 067525eb27a1d6c65682a1db3dfa0990 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/00280a66-eb0b-45b2-a2fe-ecab328454d4_biasIsr_1123733_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/00280a66-eb0b-45b2-a2fe-ecab328454d4_biasIsr_1123733_43.stderr parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 4c0cfcd989ded0dd8c59fecea411e071 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f148c0a3-3ce4-4a27-afb4-ea70d90be86a_biasIsr_1123836_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f148c0a3-3ce4-4a27-afb4-ea70d90be86a_biasIsr_1123836_43.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash ec1d44070ad1685daad51a72a590413a parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/e37e59d3-d266-42e1-8fb9-6ae7d8e97537_biasIsr_1123735_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/e37e59d3-d266-42e1-8fb9-6ae7d8e97537_biasIsr_1123735_43.stderr parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 94b79cc26d9b8bd3ecc011c3a074802a parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bca340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e2700609-58e3-4325-9e6f-ef486d0280c3_biasIsr_1123728_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e2700609-58e3-4325-9e6f-ef486d0280c3_biasIsr_1123728_43.stderr parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash bdd8c1536293ea55df77f72bb343834f parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/80f2087d-a52d-4ee9-a91b-ef9b4c64972e_biasIsr_1123737_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/80f2087d-a52d-4ee9-a91b-ef9b4c64972e_biasIsr_1123737_43.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 22094c1caa130c8275add7a3216dcafa parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/5478d10a-b055-446e-a6e8-e0f5cbaf43df_biasIsr_1123839_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/5478d10a-b055-446e-a6e8-e0f5cbaf43df_biasIsr_1123839_43.stderr parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash a8f39630effcebb41e2cb57789dbecbd parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f5dff12a-f571-44af-9217-f712cd44d3d5_biasIsr_1123830_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f5dff12a-f571-44af-9217-f712cd44d3d5_biasIsr_1123830_43.stderr parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash c3dbe24a24f94e4afa81542d68bb3e45 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3049ea3b-4358-4c28-966c-09e65cb0be2e_biasIsr_1123731_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/3049ea3b-4358-4c28-966c-09e65cb0be2e_biasIsr_1123731_43.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash baa26a81a3d1dd7d12af4faead12d588 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6011a530-321f-4c98-bbed-4284f9f4a49f_biasIsr_1123829_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6011a530-321f-4c98-bbed-4284f9f4a49f_biasIsr_1123829_43.stderr parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 5b7c8af1498b3f918b8cde7959bc7f78 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/e3a4d6b3-9c5b-4e9c-9fe8-bbc251bdcfd8_biasIsr_1123837_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/e3a4d6b3-9c5b-4e9c-9fe8-bbc251bdcfd8_biasIsr_1123837_43.stderr parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 8f89872d8120e4df9758b2879ec116b9 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b091c0b3-5d56-4d5a-9cde-9d3d53376d4d_biasIsr_1123730_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b091c0b3-5d56-4d5a-9cde-9d3d53376d4d_biasIsr_1123730_43.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 356834a1fc3494b2c686e0d21547f5f2 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/7c0f7682-395f-4dae-81c1-dd6019f0603d_biasIsr_1123831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/7c0f7682-395f-4dae-81c1-dd6019f0603d_biasIsr_1123831_43.stderr parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 74866a5cfa3417f04dda88754094e8ee parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/04fb1dc3-15aa-4900-9cad-a488714fa25f_biasIsr_1123833_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/04fb1dc3-15aa-4900-9cad-a488714fa25f_biasIsr_1123833_43.stderr parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 9872b9c748ff54f77500558d90209100 parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/f876a19a-95e1-4f45-ae52-fa23aefaf484_biasIsr_1123832_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/f876a19a-95e1-4f45-ae52-fa23aefaf484_biasIsr_1123832_43.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash db08bad4a49ca41fcfd8d6c47528ad9f parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/70dfebf8-b679-4967-b869-4413b688e985_biasIsr_1123732_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/70dfebf8-b679-4967-b869-4413b688e985_biasIsr_1123732_43.stderr parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash eb07f2a5dfe922140760e78e7021e41a parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/344464ee-0f88-421c-9216-5a04aa237da8_biasIsr_1123835_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/344464ee-0f88-421c-9216-5a04aa237da8_biasIsr_1123835_43.stderr parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash e75f8656560b9acb8c768d6eb0b0a798 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/5c1f5d12-4645-410b-94f9-f0ce5e4b72b3_biasIsr_1123738_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/5c1f5d12-4645-410b-94f9-f0ce5e4b72b3_biasIsr_1123738_43.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash 9a47c1a73afa30a04b79e971c0116e7e parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/88932984-72d3-4ca9-839c-54b9d4721701_biasIsr_1123734_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/88932984-72d3-4ca9-839c-54b9d4721701_biasIsr_1123734_43.stderr parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash cac6a7c166cd99c054cfdc6b140def59 parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ef2ae5cb-d33e-4bd1-adea-655713ee71db_biasIsr_1123834_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/ef2ae5cb-d33e-4bd1-adea-655713ee71db_biasIsr_1123834_43.stderr parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 15c0474e556ae38fe9e490020f8362ad parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/3df7bb4a-2b85-4c6c-9a7a-73d2bf022e11_biasIsr_1123729_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/3df7bb4a-2b85-4c6c-9a7a-73d2bf022e11_biasIsr_1123729_43.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash db751f1903d511f79cd2bda151912ceb parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9bcbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/929efd4d-dfde-4ded-af9c-295fe85c01c2_biasIsr_1123736_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/929efd4d-dfde-4ded-af9c-295fe85c01c2_biasIsr_1123736_43.stderr parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App cpBiasCombine, waiting on task 1081, task 1082, task 1083, task 1084, task 1085, task 1086, task 1087, task 1088, task 1089, task 1090, task 1091, task 1092, task 1093, task 1094, task 1095, task 1096, task 1097, task 1098, task 1099, task 1100, task 1101, task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 24ee4f586917eb306d9129f8369b2902 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0cb82a28-17af-495f-a1ad-b1c7d5c06b4a_biasIsr_1123732_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0cb82a28-17af-495f-a1ad-b1c7d5c06b4a_biasIsr_1123732_4.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 18af8fa2ea25eb954cc54c415836b633 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ac755742-9cf0-4ef1-96d9-372463e7c018_biasIsr_1123730_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/ac755742-9cf0-4ef1-96d9-372463e7c018_biasIsr_1123730_4.stderr parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 9bf494a1bbc4bba0b4a5ebb8e8cab2fb parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6fdde130-7454-480e-9167-e413c9b7a24d_biasIsr_1123829_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/6fdde130-7454-480e-9167-e413c9b7a24d_biasIsr_1123829_4.stderr parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash e8963a638d3f10fb2b30b9e493a2d8fe parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c88d0461-5d68-4126-b8ab-26ee9336549d_biasIsr_1123833_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c88d0461-5d68-4126-b8ab-26ee9336549d_biasIsr_1123833_4.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 2a176a36d864267a85d65f29798eaf29 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/67feb2d6-6b75-4716-92aa-bd4a1ad4be1b_biasIsr_1123830_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/67feb2d6-6b75-4716-92aa-bd4a1ad4be1b_biasIsr_1123830_4.stderr parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 0c6c9633153fd05b4485d0fb729fb3b8 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/5d651339-7cb1-48b7-877f-a14dc98885b7_biasIsr_1123834_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/5d651339-7cb1-48b7-877f-a14dc98885b7_biasIsr_1123834_4.stderr parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash bed7b2868bb656d841244e29a920c560 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c20fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5bb40446-acb9-4e94-84bb-556f6ddf794e_biasIsr_1123731_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5bb40446-acb9-4e94-84bb-556f6ddf794e_biasIsr_1123731_4.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 41ecdeaaeee12c4703304347cee6b655 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7c04415f-9a40-4035-931b-767d0b265af9_biasIsr_1123832_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/7c04415f-9a40-4035-931b-767d0b265af9_biasIsr_1123832_4.stderr parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 35637ea433fe56857cc1e8bc75d9d2d7 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e50dd7a7-36c3-487b-be7f-2f9f6d3e4373_biasIsr_1123831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e50dd7a7-36c3-487b-be7f-2f9f6d3e4373_biasIsr_1123831_4.stderr parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash e87d9017e6f461b6af87679d6c980216 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c5458c22-879f-4e89-bcb2-fd6859ed7785_biasIsr_1123835_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c5458c22-879f-4e89-bcb2-fd6859ed7785_biasIsr_1123835_4.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash ef63395487f0764f080c24aaa6ceff5e parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a073a50f-e22e-4859-b3a7-ab4e1d2cbf62_biasIsr_1123836_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a073a50f-e22e-4859-b3a7-ab4e1d2cbf62_biasIsr_1123836_4.stderr parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 838d744f5cf203f68c87ba66b0141038 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/42c17da0-0c77-4512-baaf-e28cbc121df5_biasIsr_1123733_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/42c17da0-0c77-4512-baaf-e28cbc121df5_biasIsr_1123733_4.stderr parsl.dataflow.dflow DEBUG: Task 1116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1116 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1116 has memoization hash 2e8a26ad51f59e70fa8fc2e2a56f2454 parsl.dataflow.memoization INFO: Task 1116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1116 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7289b23c-b4af-4ac4-9bf1-b42a6fbad79d_biasIsr_1123729_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/7289b23c-b4af-4ac4-9bf1-b42a6fbad79d_biasIsr_1123729_4.stderr parsl.dataflow.dflow DEBUG: Task 1117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1117 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1117 has memoization hash 331bb3d9fad024dec56adce5ab80d85a parsl.dataflow.memoization INFO: Task 1117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1117 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/62dab36c-0981-44f4-80a5-eb6a63384893_biasIsr_1123837_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/62dab36c-0981-44f4-80a5-eb6a63384893_biasIsr_1123837_4.stderr parsl.dataflow.dflow DEBUG: Task 1118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1118 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1118 has memoization hash 6d6ae27dbf08d4a23d0910e554f647bb parsl.dataflow.memoization INFO: Task 1118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1118 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/96b158be-f758-43f2-98b0-ad6d0e518794_biasIsr_1123738_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/96b158be-f758-43f2-98b0-ad6d0e518794_biasIsr_1123738_4.stderr parsl.dataflow.dflow DEBUG: Task 1119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1119 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1119 has memoization hash 812ca16dd5b85b24f60c2907acb91897 parsl.dataflow.memoization INFO: Task 1119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1119 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/df4ab6d9-bf11-4b19-ba72-20d459961e63_biasIsr_1123736_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/df4ab6d9-bf11-4b19-ba72-20d459961e63_biasIsr_1123736_4.stderr parsl.dataflow.dflow DEBUG: Task 1120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1120 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1120 has memoization hash 25a59535a85a0da7addea787356d3ce0 parsl.dataflow.memoization INFO: Task 1120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1120 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/bf59f566-ab44-4e61-8a1d-c3a0cfebdd61_biasIsr_1123838_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/bf59f566-ab44-4e61-8a1d-c3a0cfebdd61_biasIsr_1123838_4.stderr parsl.dataflow.dflow DEBUG: Task 1121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1121 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1121 has memoization hash 75b3a3988cb29319d4c353fa275a84a2 parsl.dataflow.memoization INFO: Task 1121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1121 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/49790330-2be3-47b4-ba24-e1383145cc1a_biasIsr_1123737_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/49790330-2be3-47b4-ba24-e1383145cc1a_biasIsr_1123737_4.stderr parsl.dataflow.dflow DEBUG: Task 1122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1122 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1122 has memoization hash c620a90fb0a86a0a1bec8b70a9487d0d parsl.dataflow.memoization INFO: Task 1122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1122 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/30b24d8f-a8ed-433c-924a-f285b8f5f8c9_biasIsr_1123728_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/30b24d8f-a8ed-433c-924a-f285b8f5f8c9_biasIsr_1123728_4.stderr parsl.dataflow.dflow DEBUG: Task 1123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1123 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1123 has memoization hash 9e8dce873adcb9875fe08d724700a943 parsl.dataflow.memoization INFO: Task 1123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1123 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/5accac68-71a7-4746-89fd-f0054a30053d_biasIsr_1123734_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/5accac68-71a7-4746-89fd-f0054a30053d_biasIsr_1123734_4.stderr parsl.dataflow.dflow DEBUG: Task 1124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1124 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1124 has memoization hash f7294fe22a5a2e95020ff9ebbb1c9aa9 parsl.dataflow.memoization INFO: Task 1124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1124 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d909aa8b-6eb6-4b37-8242-1261b289794b_biasIsr_1123735_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d909aa8b-6eb6-4b37-8242-1261b289794b_biasIsr_1123735_4.stderr parsl.dataflow.dflow DEBUG: Task 1125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1125 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1125 has memoization hash a5c3a609bc98f55c2b661aaf975f6b5b parsl.dataflow.memoization INFO: Task 1125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1125 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/6533dc9c-5264-4678-9b66-d745ab175935_biasIsr_1123839_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/6533dc9c-5264-4678-9b66-d745ab175935_biasIsr_1123839_4.stderr parsl.dataflow.dflow DEBUG: Task 1126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1126 submitted for App cpBiasCombine, waiting on task 1104, task 1105, task 1106, task 1107, task 1108, task 1109, task 1110, task 1111, task 1112, task 1113, task 1114, task 1115, task 1116, task 1117, task 1118, task 1119, task 1120, task 1121, task 1122, task 1123, task 1124, task 1125 parsl.dataflow.dflow DEBUG: Task 1126 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1126 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1127 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1127 has memoization hash 206d29ddc5605a26b272dd7bcc0b3687 parsl.dataflow.memoization INFO: Task 1127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1127 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/263b2e88-a5cb-4b95-b117-8f853427babc_biasIsr_1123732_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/263b2e88-a5cb-4b95-b117-8f853427babc_biasIsr_1123732_17.stderr parsl.dataflow.dflow DEBUG: Task 1128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1128 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1128 has memoization hash 49914dc395fc7ca894d81a0ef1cec881 parsl.dataflow.memoization INFO: Task 1128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1128 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a77a58a2-50ab-4ff4-ae7e-ba419335d62f_biasIsr_1123736_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/a77a58a2-50ab-4ff4-ae7e-ba419335d62f_biasIsr_1123736_17.stderr parsl.dataflow.dflow DEBUG: Task 1129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1129 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1129 has memoization hash 284ee59cda2491688e26dd86e4ab8800 parsl.dataflow.memoization INFO: Task 1129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1129 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/2827ae93-4556-4ee2-abae-b4dbc860820d_biasIsr_1123834_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/2827ae93-4556-4ee2-abae-b4dbc860820d_biasIsr_1123834_17.stderr parsl.dataflow.dflow DEBUG: Task 1130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1130 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1130 has memoization hash 73e6751e24e899cf656f8691115f7091 parsl.dataflow.memoization INFO: Task 1130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c21bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1130 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f66f6469-6601-4115-a291-83635993da34_biasIsr_1123738_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f66f6469-6601-4115-a291-83635993da34_biasIsr_1123738_17.stderr parsl.dataflow.dflow DEBUG: Task 1131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1131 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1131 has memoization hash cdae38837adf89c82e69f84c3a1cd6f2 parsl.dataflow.memoization INFO: Task 1131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1131 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/9e92893e-890c-41bc-850a-3be7667c9547_biasIsr_1123837_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/9e92893e-890c-41bc-850a-3be7667c9547_biasIsr_1123837_17.stderr parsl.dataflow.dflow DEBUG: Task 1132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1132 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1132 has memoization hash 0881c91c53bb85fca59f59e5bf899af5 parsl.dataflow.memoization INFO: Task 1132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1132 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/da195f24-09bb-4692-9e62-456e8df55d4b_biasIsr_1123832_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/da195f24-09bb-4692-9e62-456e8df55d4b_biasIsr_1123832_17.stderr parsl.dataflow.dflow DEBUG: Task 1133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1133 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1133 has memoization hash 1935921e0e6c02169254af2274b3681d parsl.dataflow.memoization INFO: Task 1133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1133 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/8b3852b5-13d8-4d87-8a82-d3d7133bd503_biasIsr_1123829_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/8b3852b5-13d8-4d87-8a82-d3d7133bd503_biasIsr_1123829_17.stderr parsl.dataflow.dflow DEBUG: Task 1134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1134 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1134 has memoization hash fc2feec8145974bfd1c4d5ccd16c2ace parsl.dataflow.memoization INFO: Task 1134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1134 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d44ddede-3be8-45fb-8207-1dae41f1b31e_biasIsr_1123734_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d44ddede-3be8-45fb-8207-1dae41f1b31e_biasIsr_1123734_17.stderr parsl.dataflow.dflow DEBUG: Task 1135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1135 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1135 has memoization hash 5e56760253e5270c38ea33a11a09371c parsl.dataflow.memoization INFO: Task 1135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1135 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/18790a74-b580-44b3-a0a3-2231d6f58733_biasIsr_1123737_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/18790a74-b580-44b3-a0a3-2231d6f58733_biasIsr_1123737_17.stderr parsl.dataflow.dflow DEBUG: Task 1136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1136 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1136 has memoization hash 63c975046808f68df2ca5d5848bd4f1d parsl.dataflow.memoization INFO: Task 1136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1136 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/a9b12716-5704-4eaf-b5c7-3659316ea47b_biasIsr_1123728_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/a9b12716-5704-4eaf-b5c7-3659316ea47b_biasIsr_1123728_17.stderr parsl.dataflow.dflow DEBUG: Task 1137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1137 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1137 has memoization hash 3e737448ef1c2919b908cfe222e6f53b parsl.dataflow.memoization INFO: Task 1137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1137 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/aac44a87-42cc-4307-a658-d95329ef66af_biasIsr_1123833_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/aac44a87-42cc-4307-a658-d95329ef66af_biasIsr_1123833_17.stderr parsl.dataflow.dflow DEBUG: Task 1138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1138 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1138 has memoization hash cb00b9cd0158c3c61c3de9d7479cda9d parsl.dataflow.memoization INFO: Task 1138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1138 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bc0bcbd1-8013-4753-b953-10a26bad24bc_biasIsr_1123839_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bc0bcbd1-8013-4753-b953-10a26bad24bc_biasIsr_1123839_17.stderr parsl.dataflow.dflow DEBUG: Task 1139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1139 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1139 has memoization hash 3467c1558ebe57c8a496148c0c6db35b parsl.dataflow.memoization INFO: Task 1139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c22fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1139 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/742d98ef-1847-49df-b23d-97d94b9762e2_biasIsr_1123735_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/742d98ef-1847-49df-b23d-97d94b9762e2_biasIsr_1123735_17.stderr parsl.dataflow.dflow DEBUG: Task 1140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1140 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1140 has memoization hash bdebcb5a7a550c26c2a4c38ee5ec24dd parsl.dataflow.memoization INFO: Task 1140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1140 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/649e3a9c-c066-4649-84f2-6e0d27ebd1c8_biasIsr_1123835_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/649e3a9c-c066-4649-84f2-6e0d27ebd1c8_biasIsr_1123835_17.stderr parsl.dataflow.dflow DEBUG: Task 1141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1141 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1141 has memoization hash 5bad88e3442ce38e4636456a581c1a57 parsl.dataflow.memoization INFO: Task 1141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1141 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a6f6de85-cd4c-422a-ad39-6e35b445f381_biasIsr_1123729_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/a6f6de85-cd4c-422a-ad39-6e35b445f381_biasIsr_1123729_17.stderr parsl.dataflow.dflow DEBUG: Task 1142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1142 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1142 has memoization hash 4e6e811064031fd10b84d5b56a21f8a2 parsl.dataflow.memoization INFO: Task 1142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1142 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/26c6556d-6d30-4768-889d-6cdb4d75198a_biasIsr_1123733_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/26c6556d-6d30-4768-889d-6cdb4d75198a_biasIsr_1123733_17.stderr parsl.dataflow.dflow DEBUG: Task 1143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1143 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1143 has memoization hash aaf9117579062099229094f961e403ac parsl.dataflow.memoization INFO: Task 1143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1143 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/12cfbcc3-7806-4562-a354-63dd22afbbcd_biasIsr_1123830_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/12cfbcc3-7806-4562-a354-63dd22afbbcd_biasIsr_1123830_17.stderr parsl.dataflow.dflow DEBUG: Task 1144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1144 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1144 has memoization hash dbe2a140fd9b6d0c2ddbb9f7350577de parsl.dataflow.memoization INFO: Task 1144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1144 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/3452fa93-a8b6-4a3b-909b-dc0659abc416_biasIsr_1123730_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/3452fa93-a8b6-4a3b-909b-dc0659abc416_biasIsr_1123730_17.stderr parsl.dataflow.dflow DEBUG: Task 1145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1145 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1145 has memoization hash 4423f0b6fbba4599afa00dfef47acc6e parsl.dataflow.memoization INFO: Task 1145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1145 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/fa10b553-0ff7-4b23-bd3c-52f3e3e3f39c_biasIsr_1123838_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/fa10b553-0ff7-4b23-bd3c-52f3e3e3f39c_biasIsr_1123838_17.stderr parsl.dataflow.dflow DEBUG: Task 1146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1146 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1146 has memoization hash 6f0e7eaa777beb6708692dd75351a874 parsl.dataflow.memoization INFO: Task 1146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1146 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/24178b6e-d9db-48d9-bafd-b1a1a383ad08_biasIsr_1123831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/24178b6e-d9db-48d9-bafd-b1a1a383ad08_biasIsr_1123831_17.stderr parsl.dataflow.dflow DEBUG: Task 1147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1147 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1147 has memoization hash 62dcba1d964345e07aed495f9a19646a parsl.dataflow.memoization INFO: Task 1147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1147 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c3b652de-f02a-4f1e-bda7-c23f0df73a09_biasIsr_1123731_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/c3b652de-f02a-4f1e-bda7-c23f0df73a09_biasIsr_1123731_17.stderr parsl.dataflow.dflow DEBUG: Task 1148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1148 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1148 has memoization hash f425390bdc370f13487d363de317b4bf parsl.dataflow.memoization INFO: Task 1148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c23560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1148 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/bd907ef5-462f-4c39-9c08-4a5f73c52d0b_biasIsr_1123836_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/bd907ef5-462f-4c39-9c08-4a5f73c52d0b_biasIsr_1123836_17.stderr parsl.dataflow.dflow DEBUG: Task 1149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1149 submitted for App cpBiasCombine, waiting on task 1127, task 1128, task 1129, task 1130, task 1131, task 1132, task 1133, task 1134, task 1135, task 1136, task 1137, task 1138, task 1139, task 1140, task 1141, task 1142, task 1143, task 1144, task 1145, task 1146, task 1147, task 1148 parsl.dataflow.dflow DEBUG: Task 1149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1150 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1150 has memoization hash aa5195e91003592002526ff9bec207a2 parsl.dataflow.memoization INFO: Task 1150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1150 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/d0503f7e-ffdf-4b2a-9c85-071c0d9a71bb_biasIsr_1123732_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/d0503f7e-ffdf-4b2a-9c85-071c0d9a71bb_biasIsr_1123732_3.stderr parsl.dataflow.dflow DEBUG: Task 1151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1151 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1151 has memoization hash 6abd1d3393db88a77e6b51f0530fcd22 parsl.dataflow.memoization INFO: Task 1151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1151 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/77b376b3-2373-4720-8822-440136e16ded_biasIsr_1123730_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/77b376b3-2373-4720-8822-440136e16ded_biasIsr_1123730_3.stderr parsl.dataflow.dflow DEBUG: Task 1152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1152 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1152 has memoization hash 22c98bcdadb2544758d654425e0fc771 parsl.dataflow.memoization INFO: Task 1152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c585e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1152 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6fe7253f-89fa-45fa-a2d1-84b4b849e5c2_biasIsr_1123833_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6fe7253f-89fa-45fa-a2d1-84b4b849e5c2_biasIsr_1123833_3.stderr parsl.dataflow.dflow DEBUG: Task 1153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1153 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1153 has memoization hash e24320a43e173980e909558c2a2febd9 parsl.dataflow.memoization INFO: Task 1153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1153 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/19eaa15a-90b4-469e-be45-73bb9e781e2f_biasIsr_1123728_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/19eaa15a-90b4-469e-be45-73bb9e781e2f_biasIsr_1123728_3.stderr parsl.dataflow.dflow DEBUG: Task 1154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1154 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1154 has memoization hash 11415a6c2b5dce6131cf90c48b4de057 parsl.dataflow.memoization INFO: Task 1154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c587c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1154 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/da7daf15-1c23-4bbb-b36f-a4027c3d50d4_biasIsr_1123731_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/da7daf15-1c23-4bbb-b36f-a4027c3d50d4_biasIsr_1123731_3.stderr parsl.dataflow.dflow DEBUG: Task 1155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1155 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1155 has memoization hash 110405f7fe256390722a59111d369e7d parsl.dataflow.memoization INFO: Task 1155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1155 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9e14a481-23c7-48a2-a987-4af1c15cd014_biasIsr_1123736_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9e14a481-23c7-48a2-a987-4af1c15cd014_biasIsr_1123736_3.stderr parsl.dataflow.dflow DEBUG: Task 1156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1156 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1156 has memoization hash 5546f56f6a5052c7e934a25402569944 parsl.dataflow.memoization INFO: Task 1156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c589a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1156 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f7228998-1fd1-400f-97a0-bc69824f50aa_biasIsr_1123836_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/f7228998-1fd1-400f-97a0-bc69824f50aa_biasIsr_1123836_3.stderr parsl.dataflow.dflow DEBUG: Task 1157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1157 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1157 has memoization hash fcb4f3165b6d2b0a92c6b37e3b992048 parsl.dataflow.memoization INFO: Task 1157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1157 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1a36f23d-dacb-4747-8106-e526b00a997c_biasIsr_1123835_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/1a36f23d-dacb-4747-8106-e526b00a997c_biasIsr_1123835_3.stderr parsl.dataflow.dflow DEBUG: Task 1158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1158 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1158 has memoization hash e4ca7883703246273e52ab464d7b790c parsl.dataflow.memoization INFO: Task 1158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1158 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7b6c2eb2-d520-46cc-bc05-b460b127b449_biasIsr_1123733_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7b6c2eb2-d520-46cc-bc05-b460b127b449_biasIsr_1123733_3.stderr parsl.dataflow.dflow DEBUG: Task 1159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1159 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1159 has memoization hash 025fd3d46f488cf98b9c047400c62701 parsl.dataflow.memoization INFO: Task 1159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1159 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/34f8c04c-8902-448d-af5b-cd108bec8789_biasIsr_1123834_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/34f8c04c-8902-448d-af5b-cd108bec8789_biasIsr_1123834_3.stderr parsl.dataflow.dflow DEBUG: Task 1160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1160 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1160 has memoization hash c2115b242456c4982685bc18e08e1910 parsl.dataflow.memoization INFO: Task 1160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1160 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/6e73f694-6829-4515-9b6b-b920f593640c_biasIsr_1123738_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/6e73f694-6829-4515-9b6b-b920f593640c_biasIsr_1123738_3.stderr parsl.dataflow.dflow DEBUG: Task 1161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1161 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1161 has memoization hash 2fdb44715c1290deb69041edb75833f0 parsl.dataflow.memoization INFO: Task 1161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1161 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/cd01ddee-730e-4450-8919-f3323bffef80_biasIsr_1123839_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/cd01ddee-730e-4450-8919-f3323bffef80_biasIsr_1123839_3.stderr parsl.dataflow.dflow DEBUG: Task 1162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1162 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1162 has memoization hash 30a18ed4f1b15bbd80831f583fe96790 parsl.dataflow.memoization INFO: Task 1162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1162 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e965238e-1334-4cd1-aaeb-e542a42e2757_biasIsr_1123734_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/e965238e-1334-4cd1-aaeb-e542a42e2757_biasIsr_1123734_3.stderr parsl.dataflow.dflow DEBUG: Task 1163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1163 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1163 has memoization hash cbabf31e2da1ad7fbbc84c0fbf54c7b3 parsl.dataflow.memoization INFO: Task 1163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1163 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d1056257-57c6-45a1-9cd0-ff0bc46542ac_biasIsr_1123729_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d1056257-57c6-45a1-9cd0-ff0bc46542ac_biasIsr_1123729_3.stderr parsl.dataflow.dflow DEBUG: Task 1164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1164 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1164 has memoization hash 4f2b60b69f963a5493aacde27fcc3101 parsl.dataflow.memoization INFO: Task 1164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1164 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f8c070ce-fa74-46e6-a533-39045dc5f852_biasIsr_1123831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/f8c070ce-fa74-46e6-a533-39045dc5f852_biasIsr_1123831_3.stderr parsl.dataflow.dflow DEBUG: Task 1165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1165 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1165 has memoization hash d5895c9d4a3fe60c2aec6621300f0c05 parsl.dataflow.memoization INFO: Task 1165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1165 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c561a59e-c726-4811-a58c-99f4a3377ed3_biasIsr_1123735_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c561a59e-c726-4811-a58c-99f4a3377ed3_biasIsr_1123735_3.stderr parsl.dataflow.dflow DEBUG: Task 1166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1166 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1166 has memoization hash cc5b7951c0a8689f6497a116bf80702e parsl.dataflow.memoization INFO: Task 1166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1166 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e7c81511-f9dd-4651-a95f-1e1d024abf5f_biasIsr_1123829_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e7c81511-f9dd-4651-a95f-1e1d024abf5f_biasIsr_1123829_3.stderr parsl.dataflow.dflow DEBUG: Task 1167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1167 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1167 has memoization hash 382783490486288e1e156e13c799dfbd parsl.dataflow.memoization INFO: Task 1167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c58f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1167 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ddb91bb0-1f33-4ea0-b86b-1469937f5e1a_biasIsr_1123838_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ddb91bb0-1f33-4ea0-b86b-1469937f5e1a_biasIsr_1123838_3.stderr parsl.dataflow.dflow DEBUG: Task 1168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1168 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1168 has memoization hash 35e0e064dee5072b12f4e891dff83877 parsl.dataflow.memoization INFO: Task 1168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c59080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1168 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/b86fb9f6-43c5-4709-b85b-ae4230450f35_biasIsr_1123832_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/b86fb9f6-43c5-4709-b85b-ae4230450f35_biasIsr_1123832_3.stderr parsl.dataflow.dflow DEBUG: Task 1169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1169 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1169 has memoization hash a54b629f44335cdb44d91dcaf40672f9 parsl.dataflow.memoization INFO: Task 1169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c59120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1169 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/865cb6c6-8302-4f67-9bb2-3e2bd17dd661_biasIsr_1123830_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/865cb6c6-8302-4f67-9bb2-3e2bd17dd661_biasIsr_1123830_3.stderr parsl.dataflow.dflow DEBUG: Task 1170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1170 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1170 has memoization hash 7f610b132a2045c9cc6cd68c341fdc60 parsl.dataflow.memoization INFO: Task 1170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c591c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1170 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b116fb52-1e2e-4b69-b94d-377df71a5276_biasIsr_1123737_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b116fb52-1e2e-4b69-b94d-377df71a5276_biasIsr_1123737_3.stderr parsl.dataflow.dflow DEBUG: Task 1171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1171 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1171 has memoization hash 4b78e47a71d182cca2d988e1aa984c0e parsl.dataflow.memoization INFO: Task 1171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c59300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1171 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a2509b06-25e2-4533-84f9-250bc32947cf_biasIsr_1123837_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/a2509b06-25e2-4533-84f9-250bc32947cf_biasIsr_1123837_3.stderr parsl.dataflow.dflow DEBUG: Task 1172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1172 submitted for App cpBiasCombine, waiting on task 1150, task 1151, task 1152, task 1153, task 1154, task 1155, task 1156, task 1157, task 1158, task 1159, task 1160, task 1161, task 1162, task 1163, task 1164, task 1165, task 1166, task 1167, task 1168, task 1169, task 1170, task 1171 parsl.dataflow.dflow DEBUG: Task 1172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1173 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1173 has memoization hash 32a8c75ab2e4c9a8240e1e96926b1983 parsl.dataflow.memoization INFO: Task 1173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1173 try 0 launched on executor multi with executor id 1123 parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b723e03d-b34c-47c8-a760-648d7b8675f0_biasIsr_1123835_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b723e03d-b34c-47c8-a760-648d7b8675f0_biasIsr_1123835_48.stderr parsl.dataflow.dflow DEBUG: Task 1174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1174 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1174 has memoization hash 472468bab5a8d1e5c2ac30bbe9585158 parsl.dataflow.memoization INFO: Task 1174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1174 try 0 launched on executor multi with executor id 1124 parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/25dfca78-6899-42a5-9dff-702930ffc559_biasIsr_1123733_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/25dfca78-6899-42a5-9dff-702930ffc559_biasIsr_1123733_48.stderr parsl.dataflow.dflow DEBUG: Task 1175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1175 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1175 has memoization hash 147f05153a07aff49a3daf21a7a7a92b parsl.dataflow.memoization INFO: Task 1175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1175 try 0 launched on executor multi with executor id 1125 parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/c6bf8ca8-3b41-4ca1-b914-9e43107df996_biasIsr_1123730_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/c6bf8ca8-3b41-4ca1-b914-9e43107df996_biasIsr_1123730_48.stderr parsl.dataflow.dflow DEBUG: Task 1176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1176 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1176 has memoization hash 80b2d9e23ab34b1e121584faecedd315 parsl.dataflow.memoization INFO: Task 1176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c59440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1176 try 0 launched on executor multi with executor id 1126 parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/0e1321d4-e241-42e5-87a0-f7595dc767bc_biasIsr_1123837_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/0e1321d4-e241-42e5-87a0-f7595dc767bc_biasIsr_1123837_48.stderr parsl.dataflow.dflow DEBUG: Task 1177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1177 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1177 has memoization hash 1e79529870cc90fda79026e44aa5718f parsl.dataflow.memoization INFO: Task 1177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1177 try 0 launched on executor multi with executor id 1127 parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/88249925-6f70-49df-9871-673eb160af5d_biasIsr_1123728_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/88249925-6f70-49df-9871-673eb160af5d_biasIsr_1123728_48.stderr parsl.dataflow.dflow DEBUG: Task 1178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1178 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1178 has memoization hash 2ae812f051bf30793abacdadce93ef0f parsl.dataflow.memoization INFO: Task 1178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1178 try 0 launched on executor multi with executor id 1128 parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/cbbee282-cd91-4750-9a39-c5050ab9f8d8_biasIsr_1123729_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/cbbee282-cd91-4750-9a39-c5050ab9f8d8_biasIsr_1123729_48.stderr parsl.dataflow.dflow DEBUG: Task 1179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1179 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1179 has memoization hash a4e528ee80adbcfa9f6bc0b60043d05e parsl.dataflow.memoization INFO: Task 1179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1179 try 0 launched on executor multi with executor id 1129 parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a4d12a29-9467-4e1e-a2a2-c76632b5cdfb_biasIsr_1123836_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a4d12a29-9467-4e1e-a2a2-c76632b5cdfb_biasIsr_1123836_48.stderr parsl.dataflow.dflow DEBUG: Task 1180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1180 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1180 has memoization hash 2720d4d889f2fa728edfd09ea97b78e1 parsl.dataflow.memoization INFO: Task 1180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1180 try 0 launched on executor multi with executor id 1130 parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/237ceb0a-2d14-4ba7-bcfa-c50dc870601c_biasIsr_1123737_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/237ceb0a-2d14-4ba7-bcfa-c50dc870601c_biasIsr_1123737_48.stderr parsl.dataflow.dflow DEBUG: Task 1181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1181 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1181 has memoization hash c8c2452b53ec8f5c631291e53aeff56b parsl.dataflow.memoization INFO: Task 1181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1181 try 0 launched on executor multi with executor id 1131 parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bbc3573f-0bad-4836-871a-572ddd980000_biasIsr_1123736_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/bbc3573f-0bad-4836-871a-572ddd980000_biasIsr_1123736_48.stderr parsl.dataflow.dflow DEBUG: Task 1182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1182 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1182 has memoization hash 3cfb3aed44c48163b59bca1bf15f0459 parsl.dataflow.memoization INFO: Task 1182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1182 try 0 launched on executor multi with executor id 1132 parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b66ca4cc-3467-4ee2-b1b1-5b02eb9b7e21_biasIsr_1123833_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/b66ca4cc-3467-4ee2-b1b1-5b02eb9b7e21_biasIsr_1123833_48.stderr parsl.dataflow.dflow DEBUG: Task 1183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1183 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1183 has memoization hash 5de59fb2587ca33dd1f497181a527f1e parsl.dataflow.memoization INFO: Task 1183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1183 try 0 launched on executor multi with executor id 1133 parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/00362b7b-e1b4-489b-aa9d-590d6216fbd6_biasIsr_1123834_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/00362b7b-e1b4-489b-aa9d-590d6216fbd6_biasIsr_1123834_48.stderr parsl.dataflow.dflow DEBUG: Task 1184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1184 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1184 has memoization hash 5805262b81c27b59b80f733d624d0445 parsl.dataflow.memoization INFO: Task 1184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1184 try 0 launched on executor multi with executor id 1134 parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a53b73ef-c85d-425d-80df-5a0108283706_biasIsr_1123735_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a53b73ef-c85d-425d-80df-5a0108283706_biasIsr_1123735_48.stderr parsl.dataflow.dflow DEBUG: Task 1185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1185 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1185 has memoization hash 71229189d6300b7d0d94854bce1b2268 parsl.dataflow.memoization INFO: Task 1185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1185 try 0 launched on executor multi with executor id 1135 parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/3daac421-ab4c-46c3-939e-fd1c9ef9620d_biasIsr_1123829_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/3daac421-ab4c-46c3-939e-fd1c9ef9620d_biasIsr_1123829_48.stderr parsl.dataflow.dflow DEBUG: Task 1186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1186 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1186 has memoization hash c78c4ebc4ab08d7a2cb6358395ab5d14 parsl.dataflow.memoization INFO: Task 1186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1186 try 0 launched on executor multi with executor id 1136 parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3aaa70d4-ec17-44e3-9125-e2afbed1a9a0_biasIsr_1123830_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/3aaa70d4-ec17-44e3-9125-e2afbed1a9a0_biasIsr_1123830_48.stderr parsl.dataflow.dflow DEBUG: Task 1187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1187 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1187 has memoization hash 3d5ba2cd914cd51c104354ac134fadc5 parsl.dataflow.memoization INFO: Task 1187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1187 try 0 launched on executor multi with executor id 1137 parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/94159b3e-10cb-4739-938e-bd12975305a1_biasIsr_1123732_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/94159b3e-10cb-4739-938e-bd12975305a1_biasIsr_1123732_48.stderr parsl.dataflow.dflow DEBUG: Task 1188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1188 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1188 has memoization hash 9761f226fa0d082665fcd303da36da22 parsl.dataflow.memoization INFO: Task 1188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1188 try 0 launched on executor multi with executor id 1138 parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b4f9e9bf-390e-4377-97d2-9499eacab446_biasIsr_1123734_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b4f9e9bf-390e-4377-97d2-9499eacab446_biasIsr_1123734_48.stderr parsl.dataflow.dflow DEBUG: Task 1189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1189 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1189 has memoization hash 6ee857dda70684a0a3c3e782bebdf528 parsl.dataflow.memoization INFO: Task 1189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1189 try 0 launched on executor multi with executor id 1139 parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/48fb77c2-bfb6-4b09-96d5-51fd9f2a9746_biasIsr_1123731_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/48fb77c2-bfb6-4b09-96d5-51fd9f2a9746_biasIsr_1123731_48.stderr parsl.dataflow.dflow DEBUG: Task 1190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1190 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1190 has memoization hash 0bb3ed1ddcdf5d460950f45c886d0425 parsl.dataflow.memoization INFO: Task 1190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1190 try 0 launched on executor multi with executor id 1140 parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/23fd8b81-e4ee-4812-b630-26eb116b9c6d_biasIsr_1123832_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/23fd8b81-e4ee-4812-b630-26eb116b9c6d_biasIsr_1123832_48.stderr parsl.dataflow.dflow DEBUG: Task 1191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1191 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1191 has memoization hash 06e2e13274d072b98399eb5c06f0b37f parsl.dataflow.memoization INFO: Task 1191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1191 try 0 launched on executor multi with executor id 1141 parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa811f45-7a6a-4f8c-84c9-e77d15526943_biasIsr_1123838_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/aa811f45-7a6a-4f8c-84c9-e77d15526943_biasIsr_1123838_48.stderr parsl.dataflow.dflow DEBUG: Task 1192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1192 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash 1bf63e971901e361eb7e01868b0ed21f parsl.dataflow.memoization INFO: Task 1192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1192 try 0 launched on executor multi with executor id 1142 parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/4d416929-08f8-4ace-aa9b-75c1e3c4e4d8_biasIsr_1123831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/4d416929-08f8-4ace-aa9b-75c1e3c4e4d8_biasIsr_1123831_48.stderr parsl.dataflow.dflow DEBUG: Task 1193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1193 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1193 has memoization hash 410ae0abe0c48051bfae4ffd411a5578 parsl.dataflow.memoization INFO: Task 1193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1193 try 0 launched on executor multi with executor id 1143 parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2dc61b6b-ff4a-40a0-8441-acd227ffb938_biasIsr_1123839_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/2dc61b6b-ff4a-40a0-8441-acd227ffb938_biasIsr_1123839_48.stderr parsl.dataflow.dflow DEBUG: Task 1194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1194 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1194 has memoization hash 278bc794f59456b1b3adc5fd7f84e04e parsl.dataflow.memoization INFO: Task 1194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1194 try 0 launched on executor multi with executor id 1144 parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/852f67f0-253f-4d2a-8170-a0468d429ceb_biasIsr_1123738_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/852f67f0-253f-4d2a-8170-a0468d429ceb_biasIsr_1123738_48.stderr parsl.dataflow.dflow DEBUG: Task 1195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1195 submitted for App cpBiasCombine, waiting on task 1173, task 1174, task 1175, task 1176, task 1177, task 1178, task 1179, task 1180, task 1181, task 1182, task 1183, task 1184, task 1185, task 1186, task 1187, task 1188, task 1189, task 1190, task 1191, task 1192, task 1193, task 1194 parsl.dataflow.dflow DEBUG: Task 1195 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1195 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1196 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1196 has memoization hash d84dc347d1ceefb2d1e5bddb70178c76 parsl.dataflow.memoization INFO: Task 1196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1196 try 0 launched on executor multi with executor id 1145 parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5c206890-3d3b-4213-8fb6-6e90c5ddfb54_biasIsr_1123731_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5c206890-3d3b-4213-8fb6-6e90c5ddfb54_biasIsr_1123731_50.stderr parsl.dataflow.dflow DEBUG: Task 1197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1197 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1197 has memoization hash 1d88fb8372273f0ed56059a8c0280dda parsl.dataflow.memoization INFO: Task 1197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1197 try 0 launched on executor multi with executor id 1146 parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/d11ddc08-41ec-442b-9296-152305adb90c_biasIsr_1123829_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/d11ddc08-41ec-442b-9296-152305adb90c_biasIsr_1123829_50.stderr parsl.dataflow.dflow DEBUG: Task 1198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1198 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 3134d82d73aaebb5e75030733e531f93 parsl.dataflow.memoization INFO: Task 1198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1198 try 0 launched on executor multi with executor id 1147 parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f211a3ba-f5ac-48a4-97a0-e817b7ed1a30_biasIsr_1123830_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/f211a3ba-f5ac-48a4-97a0-e817b7ed1a30_biasIsr_1123830_50.stderr parsl.dataflow.dflow DEBUG: Task 1199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1199 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1199 has memoization hash ccc53fa662d04ec989a47e4e26d0b33b parsl.dataflow.memoization INFO: Task 1199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1199 try 0 launched on executor multi with executor id 1148 parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7d5c6641-c22a-4e28-8575-e55d8b0950d8_biasIsr_1123738_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/7d5c6641-c22a-4e28-8575-e55d8b0950d8_biasIsr_1123738_50.stderr parsl.dataflow.dflow DEBUG: Task 1200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1200 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1200 has memoization hash baab49ff7cdac18f3a961cfc80e5a115 parsl.dataflow.memoization INFO: Task 1200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9c5be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1200 try 0 launched on executor multi with executor id 1149 parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ca59aff5-a756-46fb-8e90-146ec58c578a_biasIsr_1123835_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ca59aff5-a756-46fb-8e90-146ec58c578a_biasIsr_1123835_50.stderr parsl.dataflow.dflow DEBUG: Task 1201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1201 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1201 has memoization hash 659e44afc7fc18efc19ca7aa87d91396 parsl.dataflow.memoization INFO: Task 1201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1201 try 0 launched on executor multi with executor id 1150 parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8ca35384-a2c9-4727-8a7a-ffdd4fe4f07d_biasIsr_1123733_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/8ca35384-a2c9-4727-8a7a-ffdd4fe4f07d_biasIsr_1123733_50.stderr parsl.dataflow.dflow DEBUG: Task 1202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1202 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1202 has memoization hash ec81d345eb8a20442a08f1a8ce5992e9 parsl.dataflow.memoization INFO: Task 1202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1202 try 0 launched on executor multi with executor id 1151 parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2f1fd0b8-b2a1-4410-be7b-fd9ee3a20f30_biasIsr_1123730_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/2f1fd0b8-b2a1-4410-be7b-fd9ee3a20f30_biasIsr_1123730_50.stderr parsl.dataflow.dflow DEBUG: Task 1203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1203 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1203 has memoization hash 79c03e791acd2fe0325d71f324ecf0c4 parsl.dataflow.memoization INFO: Task 1203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1203 try 0 launched on executor multi with executor id 1152 parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4356e8d5-2e37-4172-95df-64d969d4bb95_biasIsr_1123834_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/4356e8d5-2e37-4172-95df-64d969d4bb95_biasIsr_1123834_50.stderr parsl.dataflow.dflow DEBUG: Task 1204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1204 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1204 has memoization hash 82e40207df21618206c1df938e80b9d0 parsl.dataflow.memoization INFO: Task 1204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1204 try 0 launched on executor multi with executor id 1153 parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5ed8c627-a98f-497f-b1bf-a32eef6fe401_biasIsr_1123836_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/5ed8c627-a98f-497f-b1bf-a32eef6fe401_biasIsr_1123836_50.stderr parsl.dataflow.dflow DEBUG: Task 1205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1205 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1205 has memoization hash ca7a06adcf9cdffd057e700ae8df8d0f parsl.dataflow.memoization INFO: Task 1205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1205 try 0 launched on executor multi with executor id 1154 parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/75de4802-c946-401e-bb83-08c4909ddbf9_biasIsr_1123728_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/75de4802-c946-401e-bb83-08c4909ddbf9_biasIsr_1123728_50.stderr parsl.dataflow.dflow DEBUG: Task 1206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1206 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1206 has memoization hash 019f70950adbde06c8265d5c1e43ff35 parsl.dataflow.memoization INFO: Task 1206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1206 try 0 launched on executor multi with executor id 1155 parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/763a2324-864f-471b-a424-60e237c7be08_biasIsr_1123732_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/763a2324-864f-471b-a424-60e237c7be08_biasIsr_1123732_50.stderr parsl.dataflow.dflow DEBUG: Task 1207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1207 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1207 has memoization hash e377078a4054d75af9b5534bf33418a5 parsl.dataflow.memoization INFO: Task 1207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1207 try 0 launched on executor multi with executor id 1156 parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b9cd4c93-2142-4ee7-98b4-44c045230df4_biasIsr_1123729_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b9cd4c93-2142-4ee7-98b4-44c045230df4_biasIsr_1123729_50.stderr parsl.dataflow.dflow DEBUG: Task 1208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1208 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 9472e831958d71d4b391d4749b1fdc8b parsl.dataflow.memoization INFO: Task 1208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1208 try 0 launched on executor multi with executor id 1157 parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1f95fbd-a663-4c2c-ae9d-64a320faa73f_biasIsr_1123831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b1f95fbd-a663-4c2c-ae9d-64a320faa73f_biasIsr_1123831_50.stderr parsl.dataflow.dflow DEBUG: Task 1209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1209 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1209 has memoization hash 0c905acda6e1f17f7f49a8abd97c8f40 parsl.dataflow.memoization INFO: Task 1209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1209 try 0 launched on executor multi with executor id 1158 parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/99ab1533-edcf-4a09-81f7-4aa3ddc306c8_biasIsr_1123737_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/99ab1533-edcf-4a09-81f7-4aa3ddc306c8_biasIsr_1123737_50.stderr parsl.dataflow.dflow DEBUG: Task 1210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1210 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1210 has memoization hash 660f8d7a0b81e18a852fb8a1b91ca74e parsl.dataflow.memoization INFO: Task 1210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1210 try 0 launched on executor multi with executor id 1159 parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e7cbe10f-5208-45b3-b7a6-8c3075cec8a2_biasIsr_1123833_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/e7cbe10f-5208-45b3-b7a6-8c3075cec8a2_biasIsr_1123833_50.stderr parsl.dataflow.dflow DEBUG: Task 1211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1211 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1211 has memoization hash df6d4e0a0d1cba9fe84f9b6afe573e00 parsl.dataflow.memoization INFO: Task 1211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1211 try 0 launched on executor multi with executor id 1160 parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/66a6efb9-f2a8-4445-abc6-ffcdf904761d_biasIsr_1123837_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/66a6efb9-f2a8-4445-abc6-ffcdf904761d_biasIsr_1123837_50.stderr parsl.dataflow.dflow DEBUG: Task 1212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1212 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1212 has memoization hash 9cd959e7ede53c23eafdb1177e729d1c parsl.dataflow.memoization INFO: Task 1212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1212 try 0 launched on executor multi with executor id 1161 parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/f7e2ce77-f5b3-4df7-bfbe-820416406d12_biasIsr_1123832_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/f7e2ce77-f5b3-4df7-bfbe-820416406d12_biasIsr_1123832_50.stderr parsl.dataflow.dflow DEBUG: Task 1213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1213 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1213 has memoization hash fdf60af17d7914fc6c1c266b86d605ad parsl.dataflow.memoization INFO: Task 1213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1213 try 0 launched on executor multi with executor id 1162 parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/72703567-f86a-43d0-9f1e-30b1dcd0cd37_biasIsr_1123838_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/72703567-f86a-43d0-9f1e-30b1dcd0cd37_biasIsr_1123838_50.stderr parsl.dataflow.dflow DEBUG: Task 1214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1214 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1214 has memoization hash eb477bf2dc2c99615afdd8b3f13d1ea6 parsl.dataflow.memoization INFO: Task 1214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1214 try 0 launched on executor multi with executor id 1163 parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/92d4994a-1970-4725-9584-e2ad259f88bb_biasIsr_1123735_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/92d4994a-1970-4725-9584-e2ad259f88bb_biasIsr_1123735_50.stderr parsl.dataflow.dflow DEBUG: Task 1215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1215 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1215 has memoization hash 249ec4c17b4e7b1aa27e2d2613b6db73 parsl.dataflow.memoization INFO: Task 1215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1215 try 0 launched on executor multi with executor id 1164 parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/3d582806-f244-4e76-89c8-aac4e133d2cb_biasIsr_1123736_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/3d582806-f244-4e76-89c8-aac4e133d2cb_biasIsr_1123736_50.stderr parsl.dataflow.dflow DEBUG: Task 1216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1216 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1216 has memoization hash ca0b257fa83b8910bd9324249e773337 parsl.dataflow.memoization INFO: Task 1216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1216 try 0 launched on executor multi with executor id 1165 parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb50aced-1e8d-4e7f-ae2a-a6fd26bd6f2d_biasIsr_1123839_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb50aced-1e8d-4e7f-ae2a-a6fd26bd6f2d_biasIsr_1123839_50.stderr parsl.dataflow.dflow DEBUG: Task 1217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1217 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1217 has memoization hash e1ab8df17baa8ac2d02fc1566e3a2408 parsl.dataflow.memoization INFO: Task 1217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1217 try 0 launched on executor multi with executor id 1166 parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/bb84c264-c9f1-453c-90c8-4ef9b2fa79c9_biasIsr_1123734_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/bb84c264-c9f1-453c-90c8-4ef9b2fa79c9_biasIsr_1123734_50.stderr parsl.dataflow.dflow DEBUG: Task 1218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1218 submitted for App cpBiasCombine, waiting on task 1196, task 1197, task 1198, task 1199, task 1200, task 1201, task 1202, task 1203, task 1204, task 1205, task 1206, task 1207, task 1208, task 1209, task 1210, task 1211, task 1212, task 1213, task 1214, task 1215, task 1216, task 1217 parsl.dataflow.dflow DEBUG: Task 1218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1219 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1219 has memoization hash e2bcf587828b80ffb62e581180510ebc parsl.dataflow.memoization INFO: Task 1219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1219 try 0 launched on executor multi with executor id 1167 parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ac62688f-46ef-47f3-869b-8f2f7ad30061_biasIsr_1123736_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/ac62688f-46ef-47f3-869b-8f2f7ad30061_biasIsr_1123736_51.stderr parsl.dataflow.dflow DEBUG: Task 1220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1220 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1220 has memoization hash 621f9e9713192346e7e2a687c2a4e3f0 parsl.dataflow.memoization INFO: Task 1220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1220 try 0 launched on executor multi with executor id 1168 parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6c0b184c-50c4-46ed-a6f6-3e4cf7c9d506_biasIsr_1123838_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/6c0b184c-50c4-46ed-a6f6-3e4cf7c9d506_biasIsr_1123838_51.stderr parsl.dataflow.dflow DEBUG: Task 1221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1221 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1221 has memoization hash 055bb52adbfce6ddf4a8f9ef23a0c2a2 parsl.dataflow.memoization INFO: Task 1221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1221 try 0 launched on executor multi with executor id 1169 parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b9a8106a-724b-4325-b8c9-042ae90cefad_biasIsr_1123829_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/b9a8106a-724b-4325-b8c9-042ae90cefad_biasIsr_1123829_51.stderr parsl.dataflow.dflow DEBUG: Task 1222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1222 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1222 has memoization hash 52787d9d40219b2becd2955ce865e194 parsl.dataflow.memoization INFO: Task 1222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1222 try 0 launched on executor multi with executor id 1170 parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/ab77b9ce-8b95-4b0c-a8f2-8c1fc1544a18_biasIsr_1123738_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/ab77b9ce-8b95-4b0c-a8f2-8c1fc1544a18_biasIsr_1123738_51.stderr parsl.dataflow.dflow DEBUG: Task 1223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1223 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1223 has memoization hash efd4b41d065fb9947554e77840ceefb4 parsl.dataflow.memoization INFO: Task 1223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1223 try 0 launched on executor multi with executor id 1171 parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b2ef1f04-9b7d-4c64-b43f-5d2f58cbbf7d_biasIsr_1123834_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/b2ef1f04-9b7d-4c64-b43f-5d2f58cbbf7d_biasIsr_1123834_51.stderr parsl.dataflow.dflow DEBUG: Task 1224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1224 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1224 has memoization hash f8293b017580de1d31b4f8f522eb2aca parsl.dataflow.memoization INFO: Task 1224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1224 try 0 launched on executor multi with executor id 1172 parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/36876099-7371-4b8a-8ed0-bffedc74051b_biasIsr_1123839_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/36876099-7371-4b8a-8ed0-bffedc74051b_biasIsr_1123839_51.stderr parsl.dataflow.dflow DEBUG: Task 1225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1225 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1225 has memoization hash ee8b0ff31c71e9e98ec30619a5bdbb4e parsl.dataflow.memoization INFO: Task 1225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1225 try 0 launched on executor multi with executor id 1173 parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/74bad184-a019-4b68-a5c4-e661bf1e4bcd_biasIsr_1123836_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/74bad184-a019-4b68-a5c4-e661bf1e4bcd_biasIsr_1123836_51.stderr parsl.dataflow.dflow DEBUG: Task 1226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1226 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1226 has memoization hash 90276685689b16fd68647f4ab2f88f84 parsl.dataflow.memoization INFO: Task 1226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1226 try 0 launched on executor multi with executor id 1174 parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7c3e81c7-8b28-4c7a-8a56-f11df6304c5c_biasIsr_1123830_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7c3e81c7-8b28-4c7a-8a56-f11df6304c5c_biasIsr_1123830_51.stderr parsl.dataflow.dflow DEBUG: Task 1227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1227 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1227 has memoization hash 97dab8c900cff17f483327584693658e parsl.dataflow.memoization INFO: Task 1227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1227 try 0 launched on executor multi with executor id 1175 parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f97d3151-0615-4886-b5f3-4b1d31b6c17b_biasIsr_1123837_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/f97d3151-0615-4886-b5f3-4b1d31b6c17b_biasIsr_1123837_51.stderr parsl.dataflow.dflow DEBUG: Task 1228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1228 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1228 has memoization hash 7535b52b9c2ca80abc925fcc26b2822e parsl.dataflow.memoization INFO: Task 1228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1228 try 0 launched on executor multi with executor id 1176 parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1b53440f-01ac-4234-be11-ed34f1931d84_biasIsr_1123730_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1b53440f-01ac-4234-be11-ed34f1931d84_biasIsr_1123730_51.stderr parsl.dataflow.dflow DEBUG: Task 1229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1229 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1229 has memoization hash 5538321c7755f392155af2ed730952e0 parsl.dataflow.memoization INFO: Task 1229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aa9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1229 try 0 launched on executor multi with executor id 1177 parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/22e248cf-72c1-440c-8cf6-7ad2429ea37d_biasIsr_1123835_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/22e248cf-72c1-440c-8cf6-7ad2429ea37d_biasIsr_1123835_51.stderr parsl.dataflow.dflow DEBUG: Task 1230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1230 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1230 has memoization hash f92184bf1757e54550bfac850fbbf077 parsl.dataflow.memoization INFO: Task 1230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1230 try 0 launched on executor multi with executor id 1178 parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/69849fab-0b1d-4e2c-9239-c7bb5f700e1a_biasIsr_1123737_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/69849fab-0b1d-4e2c-9239-c7bb5f700e1a_biasIsr_1123737_51.stderr parsl.dataflow.dflow DEBUG: Task 1231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1231 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1231 has memoization hash ace9a8d6dd9821ac318687757f3e0696 parsl.dataflow.memoization INFO: Task 1231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1231 try 0 launched on executor multi with executor id 1179 parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c52c54e7-4fc6-4917-9405-85fbec35f053_biasIsr_1123732_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/c52c54e7-4fc6-4917-9405-85fbec35f053_biasIsr_1123732_51.stderr parsl.dataflow.dflow DEBUG: Task 1232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1232 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1232 has memoization hash eb9bc54353eabaf73cddd6b871174d61 parsl.dataflow.memoization INFO: Task 1232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1232 try 0 launched on executor multi with executor id 1180 parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6d98c9ba-6996-41ce-a374-78969a149cc5_biasIsr_1123833_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/6d98c9ba-6996-41ce-a374-78969a149cc5_biasIsr_1123833_51.stderr parsl.dataflow.dflow DEBUG: Task 1233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1233 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1233 has memoization hash 1700e4f0d3616caa49065b81e8f6d488 parsl.dataflow.memoization INFO: Task 1233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1233 try 0 launched on executor multi with executor id 1181 parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/b9e963f5-f98c-4313-865a-9257d57fa932_biasIsr_1123735_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/b9e963f5-f98c-4313-865a-9257d57fa932_biasIsr_1123735_51.stderr parsl.dataflow.dflow DEBUG: Task 1234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1234 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1234 has memoization hash ab481dfd2cf5f2c0b04b2c5476732891 parsl.dataflow.memoization INFO: Task 1234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1234 try 0 launched on executor multi with executor id 1182 parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/c05ff51b-bef1-4ff3-9c56-36d07ab33cfb_biasIsr_1123733_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/c05ff51b-bef1-4ff3-9c56-36d07ab33cfb_biasIsr_1123733_51.stderr parsl.dataflow.dflow DEBUG: Task 1235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1235 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1235 has memoization hash 271bce86623fa10e398589c22238be9f parsl.dataflow.memoization INFO: Task 1235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1235 try 0 launched on executor multi with executor id 1183 parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/28e9c2d7-4a71-40bd-a428-e1d7a0a3f1d8_biasIsr_1123832_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/28e9c2d7-4a71-40bd-a428-e1d7a0a3f1d8_biasIsr_1123832_51.stderr parsl.dataflow.dflow DEBUG: Task 1236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1236 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1236 has memoization hash a5ec2acc30625708611027c2159ed2a4 parsl.dataflow.memoization INFO: Task 1236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1236 try 0 launched on executor multi with executor id 1184 parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6547bc3d-feb7-47e3-a297-4b9cf952b79b_biasIsr_1123734_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/6547bc3d-feb7-47e3-a297-4b9cf952b79b_biasIsr_1123734_51.stderr parsl.dataflow.dflow DEBUG: Task 1237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1237 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1237 has memoization hash 0f842410a5d36342a441f59db4cec739 parsl.dataflow.memoization INFO: Task 1237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1237 try 0 launched on executor multi with executor id 1185 parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/34c30ce7-2b3f-46ba-a63d-60fe63064be7_biasIsr_1123731_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/34c30ce7-2b3f-46ba-a63d-60fe63064be7_biasIsr_1123731_51.stderr parsl.dataflow.dflow DEBUG: Task 1238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1238 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1238 has memoization hash 037fc0ce797e9f1ac968f9549d200b5a parsl.dataflow.memoization INFO: Task 1238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1238 try 0 launched on executor multi with executor id 1186 parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/f5d0178d-c02e-477c-9337-72b287684419_biasIsr_1123729_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/f5d0178d-c02e-477c-9337-72b287684419_biasIsr_1123729_51.stderr parsl.dataflow.dflow DEBUG: Task 1239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1239 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1239 has memoization hash ee3067c555dca7feb98c4ccab1474950 parsl.dataflow.memoization INFO: Task 1239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1239 try 0 launched on executor multi with executor id 1187 parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/2901343e-c588-49cd-b53e-cfe437070d61_biasIsr_1123831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/2901343e-c588-49cd-b53e-cfe437070d61_biasIsr_1123831_51.stderr parsl.dataflow.dflow DEBUG: Task 1240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1240 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1240 has memoization hash 103573e6f5f6f837dd273b8db996656f parsl.dataflow.memoization INFO: Task 1240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1240 try 0 launched on executor multi with executor id 1188 parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8ab0dee3-7a67-4596-902f-8e6eb88fb3b2_biasIsr_1123728_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8ab0dee3-7a67-4596-902f-8e6eb88fb3b2_biasIsr_1123728_51.stderr parsl.dataflow.dflow DEBUG: Task 1241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1241 submitted for App cpBiasCombine, waiting on task 1219, task 1220, task 1221, task 1222, task 1223, task 1224, task 1225, task 1226, task 1227, task 1228, task 1229, task 1230, task 1231, task 1232, task 1233, task 1234, task 1235, task 1236, task 1237, task 1238, task 1239, task 1240 parsl.dataflow.dflow DEBUG: Task 1241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1242 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1242 has memoization hash 174aecd5831f7d314f8b5372ae8c2c12 parsl.dataflow.memoization INFO: Task 1242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1242 try 0 launched on executor multi with executor id 1189 parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f8fd6830-80fa-45e7-80f3-5bf147f7cdb2_biasIsr_1123732_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/f8fd6830-80fa-45e7-80f3-5bf147f7cdb2_biasIsr_1123732_52.stderr parsl.dataflow.dflow DEBUG: Task 1243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1243 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1243 has memoization hash d91224a2abec06f2cb78f20e9d208962 parsl.dataflow.memoization INFO: Task 1243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1243 try 0 launched on executor multi with executor id 1190 parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c9cb6dd7-5faf-401e-ba76-cef5eac930e6_biasIsr_1123735_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/c9cb6dd7-5faf-401e-ba76-cef5eac930e6_biasIsr_1123735_52.stderr parsl.dataflow.dflow DEBUG: Task 1244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1244 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1244 has memoization hash 978c15a363bbc25bfa0629384b15e3b4 parsl.dataflow.memoization INFO: Task 1244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1244 try 0 launched on executor multi with executor id 1191 parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e0ddb96e-a8d7-4df8-89a2-e1d2c81b12de_biasIsr_1123832_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/e0ddb96e-a8d7-4df8-89a2-e1d2c81b12de_biasIsr_1123832_52.stderr parsl.dataflow.dflow DEBUG: Task 1245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1245 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1245 has memoization hash e3d3b22bd855ba78f11a85f8314e1f3e parsl.dataflow.memoization INFO: Task 1245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1245 try 0 launched on executor multi with executor id 1192 parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/895c6ea0-8a31-4ed3-b6f8-fe670747e6fa_biasIsr_1123829_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/895c6ea0-8a31-4ed3-b6f8-fe670747e6fa_biasIsr_1123829_52.stderr parsl.dataflow.dflow DEBUG: Task 1246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1246 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1246 has memoization hash 8a29e2a206beb2bf274db9082ac680f0 parsl.dataflow.memoization INFO: Task 1246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1246 try 0 launched on executor multi with executor id 1193 parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/ab5b48df-b1d6-435e-bea3-98a6e3954e08_biasIsr_1123733_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/ab5b48df-b1d6-435e-bea3-98a6e3954e08_biasIsr_1123733_52.stderr parsl.dataflow.dflow DEBUG: Task 1247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1247 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1247 has memoization hash 3aac66b97322cb14e5b8dba97090e309 parsl.dataflow.memoization INFO: Task 1247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1247 try 0 launched on executor multi with executor id 1194 parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d4cb25a1-97e4-409f-a10d-7c5a365d291a_biasIsr_1123729_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/d4cb25a1-97e4-409f-a10d-7c5a365d291a_biasIsr_1123729_52.stderr parsl.dataflow.dflow DEBUG: Task 1248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1248 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1248 has memoization hash a9945f3220c9157300ab2c0c8ff52f95 parsl.dataflow.memoization INFO: Task 1248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1248 try 0 launched on executor multi with executor id 1195 parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f27f0bfd-9db0-40cd-a880-8480933c1539_biasIsr_1123738_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/f27f0bfd-9db0-40cd-a880-8480933c1539_biasIsr_1123738_52.stderr parsl.dataflow.dflow DEBUG: Task 1249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1249 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1249 has memoization hash 6bea73e52afa4a1f9fe98833bafe552a parsl.dataflow.memoization INFO: Task 1249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1249 try 0 launched on executor multi with executor id 1196 parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/0f51b130-2983-4688-aea0-05fd6c7e1f2d_biasIsr_1123837_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/0f51b130-2983-4688-aea0-05fd6c7e1f2d_biasIsr_1123837_52.stderr parsl.dataflow.dflow DEBUG: Task 1250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1250 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1250 has memoization hash 7cfe6368f275cdf5e2f61f3fc6f847d7 parsl.dataflow.memoization INFO: Task 1250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aaba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1250 try 0 launched on executor multi with executor id 1197 parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ddbfd0e7-6bd3-43de-8566-32bb5dd340ee_biasIsr_1123835_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/ddbfd0e7-6bd3-43de-8566-32bb5dd340ee_biasIsr_1123835_52.stderr parsl.dataflow.dflow DEBUG: Task 1251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1251 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1251 has memoization hash 3d5291a8d98b8dc8f4134405cda687d8 parsl.dataflow.memoization INFO: Task 1251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1251 try 0 launched on executor multi with executor id 1198 parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/983829e2-2aa1-45f4-9721-11508bfd5962_biasIsr_1123736_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/983829e2-2aa1-45f4-9721-11508bfd5962_biasIsr_1123736_52.stderr parsl.dataflow.dflow DEBUG: Task 1252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1252 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1252 has memoization hash daf147494df7fe275d60170a6a3ec13f parsl.dataflow.memoization INFO: Task 1252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1252 try 0 launched on executor multi with executor id 1199 parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/2b0e3d03-822b-489c-9385-b7b14b5080e3_biasIsr_1123838_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/2b0e3d03-822b-489c-9385-b7b14b5080e3_biasIsr_1123838_52.stderr parsl.dataflow.dflow DEBUG: Task 1253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1253 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1253 has memoization hash a8c3f14400d8cdf7a6c34758eafdb33c parsl.dataflow.memoization INFO: Task 1253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1253 try 0 launched on executor multi with executor id 1200 parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1d5cf670-a1fa-4e0f-9797-2c57f747e36b_biasIsr_1123831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/1d5cf670-a1fa-4e0f-9797-2c57f747e36b_biasIsr_1123831_52.stderr parsl.dataflow.dflow DEBUG: Task 1254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1254 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1254 has memoization hash 4b735a5a5c4d7858e98291e420cddb21 parsl.dataflow.memoization INFO: Task 1254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1254 try 0 launched on executor multi with executor id 1201 parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1c35a9e9-1424-42eb-b4c7-b394fa22b553_biasIsr_1123730_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/1c35a9e9-1424-42eb-b4c7-b394fa22b553_biasIsr_1123730_52.stderr parsl.dataflow.dflow DEBUG: Task 1255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1255 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1255 has memoization hash 563bb7db9e7482ddfcfe5f1545699db1 parsl.dataflow.memoization INFO: Task 1255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1255 try 0 launched on executor multi with executor id 1202 parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5741f7d3-66d9-4de6-b37a-e6fdcd11c968_biasIsr_1123731_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/5741f7d3-66d9-4de6-b37a-e6fdcd11c968_biasIsr_1123731_52.stderr parsl.dataflow.dflow DEBUG: Task 1256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1256 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1256 has memoization hash c86f8be78ae21e3e4e44825cdb2217d9 parsl.dataflow.memoization INFO: Task 1256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1256 try 0 launched on executor multi with executor id 1203 parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/099c2b81-f3e3-42cf-8e0d-5fd9b0c8a8e3_biasIsr_1123737_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/099c2b81-f3e3-42cf-8e0d-5fd9b0c8a8e3_biasIsr_1123737_52.stderr parsl.dataflow.dflow DEBUG: Task 1257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1257 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1257 has memoization hash 1fc355b0d12745985d7c903cf0b4d93a parsl.dataflow.memoization INFO: Task 1257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aabe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1257 try 0 launched on executor multi with executor id 1204 parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1a5899d5-35ec-4691-9ca4-00c1eb2ecb18_biasIsr_1123836_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/1a5899d5-35ec-4691-9ca4-00c1eb2ecb18_biasIsr_1123836_52.stderr parsl.dataflow.dflow DEBUG: Task 1258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1258 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1258 has memoization hash 25902b84d2d1e3fb9a77230139e51617 parsl.dataflow.memoization INFO: Task 1258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9aab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1258 try 0 launched on executor multi with executor id 1205 parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/49348b19-2744-4ad2-aa52-f369475e8b58_biasIsr_1123728_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/49348b19-2744-4ad2-aa52-f369475e8b58_biasIsr_1123728_52.stderr parsl.dataflow.dflow DEBUG: Task 1259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1259 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1259 has memoization hash 362b2d7a468b00b7ae316276c274fdcd parsl.dataflow.memoization INFO: Task 1259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b000e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1259 try 0 launched on executor multi with executor id 1206 parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/7b6b1c77-1081-46f6-a45f-51f5a06bbd84_biasIsr_1123734_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/7b6b1c77-1081-46f6-a45f-51f5a06bbd84_biasIsr_1123734_52.stderr parsl.dataflow.dflow DEBUG: Task 1260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1260 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1260 has memoization hash 5959125501c7638335c03b500ec93a8a parsl.dataflow.memoization INFO: Task 1260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b00220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1260 try 0 launched on executor multi with executor id 1207 parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/52d2a4e4-3a21-412c-9e9d-504fafcd8e2e_biasIsr_1123839_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/52d2a4e4-3a21-412c-9e9d-504fafcd8e2e_biasIsr_1123839_52.stderr parsl.dataflow.dflow DEBUG: Task 1261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1261 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1261 has memoization hash 752bda4243526d96e354e6177dadf00b parsl.dataflow.memoization INFO: Task 1261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b00180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1261 try 0 launched on executor multi with executor id 1208 parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7ebfc33e-299f-4138-9c44-d9a2f959bef5_biasIsr_1123830_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/7ebfc33e-299f-4138-9c44-d9a2f959bef5_biasIsr_1123830_52.stderr parsl.dataflow.dflow DEBUG: Task 1262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1262 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1262 has memoization hash a889695851ebbc043a7d9a1682e36e0e parsl.dataflow.memoization INFO: Task 1262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1262 try 0 launched on executor multi with executor id 1209 parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/79b42c9e-8660-4832-9dbf-a296ca607c09_biasIsr_1123834_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/79b42c9e-8660-4832-9dbf-a296ca607c09_biasIsr_1123834_52.stderr parsl.dataflow.dflow DEBUG: Task 1263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1263 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1263 has memoization hash d9f6d3982b847c82845c55204fd023b0 parsl.dataflow.memoization INFO: Task 1263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b00360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1263 try 0 launched on executor multi with executor id 1210 parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c1393319-9c25-4477-a10e-fd3bb7fe9a1c_biasIsr_1123833_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c1393319-9c25-4477-a10e-fd3bb7fe9a1c_biasIsr_1123833_52.stderr parsl.dataflow.dflow DEBUG: Task 1264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1264 submitted for App cpBiasCombine, waiting on task 1242, task 1243, task 1244, task 1245, task 1246, task 1247, task 1248, task 1249, task 1250, task 1251, task 1252, task 1253, task 1254, task 1255, task 1256, task 1257, task 1258, task 1259, task 1260, task 1261, task 1262, task 1263 parsl.dataflow.dflow DEBUG: Task 1264 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1264 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1265 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1265 has memoization hash 443e56a9027fa61da2967e8e5582bc9b parsl.dataflow.memoization INFO: Task 1265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1265 try 0 launched on executor multi with executor id 1211 parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/36c21ad9-e819-41ff-8a1d-bab4bc144f88_biasIsr_1123829_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/36c21ad9-e819-41ff-8a1d-bab4bc144f88_biasIsr_1123829_26.stderr parsl.dataflow.dflow DEBUG: Task 1266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1266 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1266 has memoization hash 2254a3fa479c7bcc6af4fe69a5f04c6c parsl.dataflow.memoization INFO: Task 1266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1266 try 0 launched on executor multi with executor id 1212 parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/57b9e242-47d2-48be-a56a-008eb8bb04b3_biasIsr_1123839_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/57b9e242-47d2-48be-a56a-008eb8bb04b3_biasIsr_1123839_26.stderr parsl.dataflow.dflow DEBUG: Task 1267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1267 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1267 has memoization hash 4d8134945561d224c75f6d57ea24940b parsl.dataflow.memoization INFO: Task 1267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1267 try 0 launched on executor multi with executor id 1213 parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/32445d4a-6480-4c2e-808a-c0a552be5ac8_biasIsr_1123731_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/32445d4a-6480-4c2e-808a-c0a552be5ac8_biasIsr_1123731_26.stderr parsl.dataflow.dflow DEBUG: Task 1268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1268 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1268 has memoization hash 9efa5dde00f11d14a33ed730f3f2fed4 parsl.dataflow.memoization INFO: Task 1268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1268 try 0 launched on executor multi with executor id 1214 parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e3580d79-d999-4bc0-ab4f-268cdf73553b_biasIsr_1123830_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e3580d79-d999-4bc0-ab4f-268cdf73553b_biasIsr_1123830_26.stderr parsl.dataflow.dflow DEBUG: Task 1269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1269 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1269 has memoization hash 36eece656ba2083b9aaf632e70eaa2bf parsl.dataflow.memoization INFO: Task 1269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1269 try 0 launched on executor multi with executor id 1215 parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/37ea658c-838d-4909-bc14-c6fcafbb0dd9_biasIsr_1123736_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/37ea658c-838d-4909-bc14-c6fcafbb0dd9_biasIsr_1123736_26.stderr parsl.dataflow.dflow DEBUG: Task 1270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1270 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1270 has memoization hash e4a5388c9b5ea175d268dd2267c10e89 parsl.dataflow.memoization INFO: Task 1270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1270 try 0 launched on executor multi with executor id 1216 parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b3b66fd9-7114-4ea4-bdfb-b9ae1ed2b1d9_biasIsr_1123836_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/b3b66fd9-7114-4ea4-bdfb-b9ae1ed2b1d9_biasIsr_1123836_26.stderr parsl.dataflow.dflow DEBUG: Task 1271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1271 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1271 has memoization hash 136cd7f1854b5f29a84cb80b9adf6c7e parsl.dataflow.memoization INFO: Task 1271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1271 try 0 launched on executor multi with executor id 1217 parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/421817d8-0a81-4141-9b93-327539e96aba_biasIsr_1123838_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/421817d8-0a81-4141-9b93-327539e96aba_biasIsr_1123838_26.stderr parsl.dataflow.dflow DEBUG: Task 1272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1272 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1272 has memoization hash bf5916528e888f58ebeea388a5eb4b68 parsl.dataflow.memoization INFO: Task 1272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1272 try 0 launched on executor multi with executor id 1218 parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/755cb580-2f65-4a29-9f0c-90f23d096545_biasIsr_1123730_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/755cb580-2f65-4a29-9f0c-90f23d096545_biasIsr_1123730_26.stderr parsl.dataflow.dflow DEBUG: Task 1273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1273 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1273 has memoization hash e5ec11c7fb1a7cadd8428769188ca009 parsl.dataflow.memoization INFO: Task 1273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1273 try 0 launched on executor multi with executor id 1219 parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a08ad2dc-679a-4518-bc93-7a3a81a29242_biasIsr_1123735_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/a08ad2dc-679a-4518-bc93-7a3a81a29242_biasIsr_1123735_26.stderr parsl.dataflow.dflow DEBUG: Task 1274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1274 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1274 has memoization hash 8cbb7bfd1695a54d2595c4f4167a0f78 parsl.dataflow.memoization INFO: Task 1274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1274 try 0 launched on executor multi with executor id 1220 parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/29fae31e-aed0-4271-aa1e-9df71c7d84a1_biasIsr_1123832_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/29fae31e-aed0-4271-aa1e-9df71c7d84a1_biasIsr_1123832_26.stderr parsl.dataflow.dflow DEBUG: Task 1275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1275 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1275 has memoization hash 67485e12eb6be2e35804dbf29d1a90ec parsl.dataflow.memoization INFO: Task 1275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1275 try 0 launched on executor multi with executor id 1221 parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/a7606907-05c9-4d80-9af9-445062810dd5_biasIsr_1123833_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/a7606907-05c9-4d80-9af9-445062810dd5_biasIsr_1123833_26.stderr parsl.dataflow.dflow DEBUG: Task 1276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1276 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1276 has memoization hash 29e65f971a10b4566280106c8cf0fd97 parsl.dataflow.memoization INFO: Task 1276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1276 try 0 launched on executor multi with executor id 1222 parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/67a6c6a4-797d-46ce-85c4-73dc6388d147_biasIsr_1123834_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/67a6c6a4-797d-46ce-85c4-73dc6388d147_biasIsr_1123834_26.stderr parsl.dataflow.dflow DEBUG: Task 1277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1277 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1277 has memoization hash 5c68c58a273405fa4741ef5086d87405 parsl.dataflow.memoization INFO: Task 1277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1277 try 0 launched on executor multi with executor id 1223 parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5eceaa91-ac71-41a1-a696-33f86ee6fa1f_biasIsr_1123728_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5eceaa91-ac71-41a1-a696-33f86ee6fa1f_biasIsr_1123728_26.stderr parsl.dataflow.dflow DEBUG: Task 1278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1278 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1278 has memoization hash 9f3f9aab6d361d2e4488cf2c74988d14 parsl.dataflow.memoization INFO: Task 1278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1278 try 0 launched on executor multi with executor id 1224 parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/6735a2b4-ae89-4f02-92d9-e3a3f112b2a8_biasIsr_1123837_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/6735a2b4-ae89-4f02-92d9-e3a3f112b2a8_biasIsr_1123837_26.stderr parsl.dataflow.dflow DEBUG: Task 1279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1279 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1279 has memoization hash 06e50416845d9de8ee86c5c1238ee5f9 parsl.dataflow.memoization INFO: Task 1279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1279 try 0 launched on executor multi with executor id 1225 parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a6e9a057-ee7a-4db2-952a-9d27aafa7ab3_biasIsr_1123733_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/a6e9a057-ee7a-4db2-952a-9d27aafa7ab3_biasIsr_1123733_26.stderr parsl.dataflow.dflow DEBUG: Task 1280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1280 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1280 has memoization hash 5b37b77645e11ab8bdf5472363ae1397 parsl.dataflow.memoization INFO: Task 1280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1280 try 0 launched on executor multi with executor id 1226 parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/38b6c795-6fd8-418c-8673-6df543ff933f_biasIsr_1123831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/38b6c795-6fd8-418c-8673-6df543ff933f_biasIsr_1123831_26.stderr parsl.dataflow.dflow DEBUG: Task 1281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1281 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1281 has memoization hash 6a281c49cdab91efae2b8258dd43768b parsl.dataflow.memoization INFO: Task 1281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1281 try 0 launched on executor multi with executor id 1227 parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/12212e4e-e150-45c4-b606-56fdf6713b14_biasIsr_1123738_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/12212e4e-e150-45c4-b606-56fdf6713b14_biasIsr_1123738_26.stderr parsl.dataflow.dflow DEBUG: Task 1282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1282 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1282 has memoization hash 8a55b557bc4e978f4f111b8a41f591c1 parsl.dataflow.memoization INFO: Task 1282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1282 try 0 launched on executor multi with executor id 1228 parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4d0fc53a-a809-49ce-8846-3477b36f4daa_biasIsr_1123729_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/4d0fc53a-a809-49ce-8846-3477b36f4daa_biasIsr_1123729_26.stderr parsl.dataflow.dflow DEBUG: Task 1283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1283 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1283 has memoization hash 506339cfa8854efed5f2586b7ceb653b parsl.dataflow.memoization INFO: Task 1283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1283 try 0 launched on executor multi with executor id 1229 parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d4b93ac2-9dd3-425c-9408-b281afae6790_biasIsr_1123734_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/d4b93ac2-9dd3-425c-9408-b281afae6790_biasIsr_1123734_26.stderr parsl.dataflow.dflow DEBUG: Task 1284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1284 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1284 has memoization hash ca972dfdf656e1f8b96158dd2c8b5d0d parsl.dataflow.memoization INFO: Task 1284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1284 try 0 launched on executor multi with executor id 1230 parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2b8c1cad-3093-4dd9-ac26-1bd5f6a3be37_biasIsr_1123737_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/2b8c1cad-3093-4dd9-ac26-1bd5f6a3be37_biasIsr_1123737_26.stderr parsl.dataflow.dflow DEBUG: Task 1285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1285 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1285 has memoization hash de413224a65df81ee6e6bd028d25cb37 parsl.dataflow.memoization INFO: Task 1285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1285 try 0 launched on executor multi with executor id 1231 parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/98588cf1-51e9-419d-9782-022bcac8d98a_biasIsr_1123732_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/98588cf1-51e9-419d-9782-022bcac8d98a_biasIsr_1123732_26.stderr parsl.dataflow.dflow DEBUG: Task 1286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1286 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1286 has memoization hash 7974cae1d3253f5bf72b23d2e9c329df parsl.dataflow.memoization INFO: Task 1286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b01ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1286 try 0 launched on executor multi with executor id 1232 parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/fbac4e80-62b9-47eb-8256-d1a01b5b71af_biasIsr_1123835_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/fbac4e80-62b9-47eb-8256-d1a01b5b71af_biasIsr_1123835_26.stderr parsl.dataflow.dflow DEBUG: Task 1287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1287 submitted for App cpBiasCombine, waiting on task 1265, task 1266, task 1267, task 1268, task 1269, task 1270, task 1271, task 1272, task 1273, task 1274, task 1275, task 1276, task 1277, task 1278, task 1279, task 1280, task 1281, task 1282, task 1283, task 1284, task 1285, task 1286 parsl.dataflow.dflow DEBUG: Task 1287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1288 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1288 has memoization hash 11b02d471b12a248288caef86fea1f6a parsl.dataflow.memoization INFO: Task 1288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b02e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1288 try 0 launched on executor multi with executor id 1233 parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/1e5c10d4-1f77-4f4d-ac2f-a9e00ba8bc69_biasIsr_1123732_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/1e5c10d4-1f77-4f4d-ac2f-a9e00ba8bc69_biasIsr_1123732_27.stderr parsl.dataflow.dflow DEBUG: Task 1289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1289 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1289 has memoization hash 228d45e532ea6a3c77102c183ad82bfa parsl.dataflow.memoization INFO: Task 1289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b02de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1289 try 0 launched on executor multi with executor id 1234 parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/c20b9ff4-7996-4d1e-962b-7c94596d8a5b_biasIsr_1123838_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/c20b9ff4-7996-4d1e-962b-7c94596d8a5b_biasIsr_1123838_27.stderr parsl.dataflow.dflow DEBUG: Task 1290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1290 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1290 has memoization hash 9a8c74eab41f93832a5ecaee166c3304 parsl.dataflow.memoization INFO: Task 1290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b02fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1290 try 0 launched on executor multi with executor id 1235 parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/576c8399-d886-413f-bf6c-cc55ef07664c_biasIsr_1123831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/576c8399-d886-413f-bf6c-cc55ef07664c_biasIsr_1123831_27.stderr parsl.dataflow.dflow DEBUG: Task 1291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1291 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1291 has memoization hash 6ed8495346979d5debe8aa930cb38900 parsl.dataflow.memoization INFO: Task 1291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b02160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1291 try 0 launched on executor multi with executor id 1236 parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/0df9532e-32d1-4a4d-a0d7-3897da4c148e_biasIsr_1123829_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/0df9532e-32d1-4a4d-a0d7-3897da4c148e_biasIsr_1123829_27.stderr parsl.dataflow.dflow DEBUG: Task 1292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1292 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1292 has memoization hash 9672507d69b12e56c218e54039c9e8da parsl.dataflow.memoization INFO: Task 1292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1292 try 0 launched on executor multi with executor id 1237 parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3c53022c-69fb-4349-8789-0606946668a7_biasIsr_1123834_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3c53022c-69fb-4349-8789-0606946668a7_biasIsr_1123834_27.stderr parsl.dataflow.dflow DEBUG: Task 1293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1293 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1293 has memoization hash 62e310c3b6918d99b9abb0a84e4799d0 parsl.dataflow.memoization INFO: Task 1293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1293 try 0 launched on executor multi with executor id 1238 parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8d9d590f-e037-4eb0-93e7-9ee4f2cbb852_biasIsr_1123836_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8d9d590f-e037-4eb0-93e7-9ee4f2cbb852_biasIsr_1123836_27.stderr parsl.dataflow.dflow DEBUG: Task 1294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1294 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1294 has memoization hash 3da10dba74a711d46a9774e780d2ef09 parsl.dataflow.memoization INFO: Task 1294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1294 try 0 launched on executor multi with executor id 1239 parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/97210da0-7aca-4655-a662-b3107877635a_biasIsr_1123833_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/97210da0-7aca-4655-a662-b3107877635a_biasIsr_1123833_27.stderr parsl.dataflow.dflow DEBUG: Task 1295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1295 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1295 has memoization hash 533895c34c107df42c7a433766793150 parsl.dataflow.memoization INFO: Task 1295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1295 try 0 launched on executor multi with executor id 1240 parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/dc1a8951-86c5-4377-8061-b5d5a30cb08f_biasIsr_1123729_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/dc1a8951-86c5-4377-8061-b5d5a30cb08f_biasIsr_1123729_27.stderr parsl.dataflow.dflow DEBUG: Task 1296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1296 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1296 has memoization hash 94f9d2521915b25b6670f863c33375cd parsl.dataflow.memoization INFO: Task 1296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1296 try 0 launched on executor multi with executor id 1241 parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/98061210-4a82-4fba-83a0-1364ff3218f0_biasIsr_1123738_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/98061210-4a82-4fba-83a0-1364ff3218f0_biasIsr_1123738_27.stderr parsl.dataflow.dflow DEBUG: Task 1297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1297 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1297 has memoization hash df38cd7a115d5eec12f6d1b3cf278df3 parsl.dataflow.memoization INFO: Task 1297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1297 try 0 launched on executor multi with executor id 1242 parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d8b425a7-0528-422c-98b2-2437e808af1a_biasIsr_1123737_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/d8b425a7-0528-422c-98b2-2437e808af1a_biasIsr_1123737_27.stderr parsl.dataflow.dflow DEBUG: Task 1298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1298 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1298 has memoization hash 13b32a688dc2f077b77c9e78113ff716 parsl.dataflow.memoization INFO: Task 1298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1298 try 0 launched on executor multi with executor id 1243 parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e6718da6-6dd7-430d-ae12-ac433975379b_biasIsr_1123728_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/e6718da6-6dd7-430d-ae12-ac433975379b_biasIsr_1123728_27.stderr parsl.dataflow.dflow DEBUG: Task 1299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1299 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1299 has memoization hash de93d8c89e6881542187f18975dc2d0e parsl.dataflow.memoization INFO: Task 1299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1299 try 0 launched on executor multi with executor id 1244 parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/edde8f54-98d8-4211-9f91-4628e9c36ab9_biasIsr_1123731_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/edde8f54-98d8-4211-9f91-4628e9c36ab9_biasIsr_1123731_27.stderr parsl.dataflow.dflow DEBUG: Task 1300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1300 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1300 has memoization hash 3847534d34d00c8e9433ed27f8faa09e parsl.dataflow.memoization INFO: Task 1300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1300 try 0 launched on executor multi with executor id 1245 parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bc1a9ff6-d11e-45ee-8e4b-5aa3aa56ee34_biasIsr_1123839_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/bc1a9ff6-d11e-45ee-8e4b-5aa3aa56ee34_biasIsr_1123839_27.stderr parsl.dataflow.dflow DEBUG: Task 1301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1301 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1301 has memoization hash 0fc368667856ae60ae74f3ac996fbd07 parsl.dataflow.memoization INFO: Task 1301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1301 try 0 launched on executor multi with executor id 1246 parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/b5205e69-9a96-4a51-842c-efc388d0000d_biasIsr_1123733_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/b5205e69-9a96-4a51-842c-efc388d0000d_biasIsr_1123733_27.stderr parsl.dataflow.dflow DEBUG: Task 1302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1302 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1302 has memoization hash 63574690bf5322c315be5521d5d1d310 parsl.dataflow.memoization INFO: Task 1302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1302 try 0 launched on executor multi with executor id 1247 parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/2f48cdf8-3791-4c3b-a97d-4b081b3ec341_biasIsr_1123830_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/2f48cdf8-3791-4c3b-a97d-4b081b3ec341_biasIsr_1123830_27.stderr parsl.dataflow.dflow DEBUG: Task 1303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1303 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1303 has memoization hash c46897c49546464ff27eb3b88f903710 parsl.dataflow.memoization INFO: Task 1303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1303 try 0 launched on executor multi with executor id 1248 parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/e6876882-b353-493b-879b-88b08ebfe4a8_biasIsr_1123730_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/e6876882-b353-493b-879b-88b08ebfe4a8_biasIsr_1123730_27.stderr parsl.dataflow.dflow DEBUG: Task 1304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1304 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1304 has memoization hash 1b237c0841708feb5d21c234a6bd1a6e parsl.dataflow.memoization INFO: Task 1304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1304 try 0 launched on executor multi with executor id 1249 parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fe924cfd-1ae7-4ca8-8828-b8236dd55a3a_biasIsr_1123832_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/fe924cfd-1ae7-4ca8-8828-b8236dd55a3a_biasIsr_1123832_27.stderr parsl.dataflow.dflow DEBUG: Task 1305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1305 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1305 has memoization hash 9056335f58a79c63a07cc32efeee8e39 parsl.dataflow.memoization INFO: Task 1305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1305 try 0 launched on executor multi with executor id 1250 parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/19132e29-d744-48a0-a864-2f66e4991888_biasIsr_1123835_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/19132e29-d744-48a0-a864-2f66e4991888_biasIsr_1123835_27.stderr parsl.dataflow.dflow DEBUG: Task 1306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1306 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1306 has memoization hash 5a8050fe4e3a28ba189abc8f3ddbff65 parsl.dataflow.memoization INFO: Task 1306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1306 try 0 launched on executor multi with executor id 1251 parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/25d9190f-942e-4b75-aad6-dab4aefdf7a6_biasIsr_1123735_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/25d9190f-942e-4b75-aad6-dab4aefdf7a6_biasIsr_1123735_27.stderr parsl.dataflow.dflow DEBUG: Task 1307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1307 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1307 has memoization hash 5c9ddcd42c9d1ae5d6be8f6a44bc0d5c parsl.dataflow.memoization INFO: Task 1307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1307 try 0 launched on executor multi with executor id 1252 parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c96b4bce-9cd6-4604-b46b-fb7cd9043c51_biasIsr_1123837_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/c96b4bce-9cd6-4604-b46b-fb7cd9043c51_biasIsr_1123837_27.stderr parsl.dataflow.dflow DEBUG: Task 1308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1308 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1308 has memoization hash cfcf3fca920d6d638a3327fa652ca519 parsl.dataflow.memoization INFO: Task 1308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1308 try 0 launched on executor multi with executor id 1253 parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/17e3ea39-2132-4cd8-8c02-b7b241c72a9e_biasIsr_1123736_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/17e3ea39-2132-4cd8-8c02-b7b241c72a9e_biasIsr_1123736_27.stderr parsl.dataflow.dflow DEBUG: Task 1309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1309 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1309 has memoization hash ac226d9d58e2c204190274bbd42be0e5 parsl.dataflow.memoization INFO: Task 1309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b03a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1309 try 0 launched on executor multi with executor id 1254 parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/a350c931-28ec-4baf-8937-16951242006d_biasIsr_1123734_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/a350c931-28ec-4baf-8937-16951242006d_biasIsr_1123734_27.stderr parsl.dataflow.dflow DEBUG: Task 1310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1310 submitted for App cpBiasCombine, waiting on task 1288, task 1289, task 1290, task 1291, task 1292, task 1293, task 1294, task 1295, task 1296, task 1297, task 1298, task 1299, task 1300, task 1301, task 1302, task 1303, task 1304, task 1305, task 1306, task 1307, task 1308, task 1309 parsl.dataflow.dflow DEBUG: Task 1310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1311 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1311 has memoization hash 0eeb20d18aee51afffe4afb4e7d6d05b parsl.dataflow.memoization INFO: Task 1311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1311 try 0 launched on executor multi with executor id 1255 parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/8cf6487a-2e1e-47d3-9cfe-a00bf4f40f46_biasIsr_1123731_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/8cf6487a-2e1e-47d3-9cfe-a00bf4f40f46_biasIsr_1123731_20.stderr parsl.dataflow.dflow DEBUG: Task 1312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1312 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1312 has memoization hash c587eae343531928cd8117bd56579f90 parsl.dataflow.memoization INFO: Task 1312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1312 try 0 launched on executor multi with executor id 1256 parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2d5cb944-37fc-45e5-904e-1dee90e4da33_biasIsr_1123728_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/2d5cb944-37fc-45e5-904e-1dee90e4da33_biasIsr_1123728_20.stderr parsl.dataflow.dflow DEBUG: Task 1313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1313 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1313 has memoization hash 6b20928cbd4a75ccf96a8137d42f380b parsl.dataflow.memoization INFO: Task 1313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1313 try 0 launched on executor multi with executor id 1257 parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b8c22067-c441-43ec-941e-be25c58c714b_biasIsr_1123729_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/b8c22067-c441-43ec-941e-be25c58c714b_biasIsr_1123729_20.stderr parsl.dataflow.dflow DEBUG: Task 1314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1314 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1314 has memoization hash 2f50339eefb5d14b0cb26fe294d12cce parsl.dataflow.memoization INFO: Task 1314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1314 try 0 launched on executor multi with executor id 1258 parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9116a89f-75ca-4609-8445-365d5e536883_biasIsr_1123736_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/9116a89f-75ca-4609-8445-365d5e536883_biasIsr_1123736_20.stderr parsl.dataflow.dflow DEBUG: Task 1315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1315 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1315 has memoization hash fabe91e50cb68ae918a5e855759e32e0 parsl.dataflow.memoization INFO: Task 1315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1315 try 0 launched on executor multi with executor id 1259 parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/4eb07557-945d-4342-a8bd-734783fe9cde_biasIsr_1123733_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/4eb07557-945d-4342-a8bd-734783fe9cde_biasIsr_1123733_20.stderr parsl.dataflow.dflow DEBUG: Task 1316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1316 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1316 has memoization hash 3fca8872b2dfe6f33915d5fc62ba969c parsl.dataflow.memoization INFO: Task 1316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1316 try 0 launched on executor multi with executor id 1260 parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/bf602f62-6926-4296-93f2-2350c6dd5509_biasIsr_1123830_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/bf602f62-6926-4296-93f2-2350c6dd5509_biasIsr_1123830_20.stderr parsl.dataflow.dflow DEBUG: Task 1317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1317 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1317 has memoization hash 6f53e0fb9cfed517927e9cb8ff8f8a77 parsl.dataflow.memoization INFO: Task 1317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1317 try 0 launched on executor multi with executor id 1261 parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b5c5d760-3f91-43cc-9501-c620ba38430e_biasIsr_1123831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b5c5d760-3f91-43cc-9501-c620ba38430e_biasIsr_1123831_20.stderr parsl.dataflow.dflow DEBUG: Task 1318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1318 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1318 has memoization hash 3d88dd21c2171168ed2b6489443a2f14 parsl.dataflow.memoization INFO: Task 1318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1318 try 0 launched on executor multi with executor id 1262 parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/2fec383e-4bc8-4284-815e-f871c28da3db_biasIsr_1123832_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/2fec383e-4bc8-4284-815e-f871c28da3db_biasIsr_1123832_20.stderr parsl.dataflow.dflow DEBUG: Task 1319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1319 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1319 has memoization hash 41d6dc4c7c2ebf366eb2e42b82a3a904 parsl.dataflow.memoization INFO: Task 1319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b50f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1319 try 0 launched on executor multi with executor id 1263 parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/0585f4fd-494d-4c2f-bd99-b40b8d40d641_biasIsr_1123730_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/0585f4fd-494d-4c2f-bd99-b40b8d40d641_biasIsr_1123730_20.stderr parsl.dataflow.dflow DEBUG: Task 1320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1320 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1320 has memoization hash 6ebdf9ec37b6fa38a8163caaeab0fff1 parsl.dataflow.memoization INFO: Task 1320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1320 try 0 launched on executor multi with executor id 1264 parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/9961fe25-b4c1-4c7c-bacc-337c535ab33b_biasIsr_1123734_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/9961fe25-b4c1-4c7c-bacc-337c535ab33b_biasIsr_1123734_20.stderr parsl.dataflow.dflow DEBUG: Task 1321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1321 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1321 has memoization hash a2ef31007c8180ee23a608563e686e6a parsl.dataflow.memoization INFO: Task 1321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1321 try 0 launched on executor multi with executor id 1265 parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/95ee035d-2b89-478b-87b4-0b38c6d68356_biasIsr_1123836_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/95ee035d-2b89-478b-87b4-0b38c6d68356_biasIsr_1123836_20.stderr parsl.dataflow.dflow DEBUG: Task 1322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1322 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1322 has memoization hash fc5c3bf804b1588e2ea5740e0e87bd2f parsl.dataflow.memoization INFO: Task 1322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1322 try 0 launched on executor multi with executor id 1266 parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d7158c42-3e44-4f81-9ceb-b68656ab9e8c_biasIsr_1123839_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d7158c42-3e44-4f81-9ceb-b68656ab9e8c_biasIsr_1123839_20.stderr parsl.dataflow.dflow DEBUG: Task 1323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1323 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1323 has memoization hash d9ccc96a4de638301fad189b0dfb5186 parsl.dataflow.memoization INFO: Task 1323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1323 try 0 launched on executor multi with executor id 1267 parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/3f9ad919-eaf8-4832-9920-c699ce17d4e7_biasIsr_1123838_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/3f9ad919-eaf8-4832-9920-c699ce17d4e7_biasIsr_1123838_20.stderr parsl.dataflow.dflow DEBUG: Task 1324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1324 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1324 has memoization hash 107ebfdfda90c53b72c33f23f3024234 parsl.dataflow.memoization INFO: Task 1324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1324 try 0 launched on executor multi with executor id 1268 parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/30b039ec-557f-4519-a755-72cf76f3b037_biasIsr_1123833_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/30b039ec-557f-4519-a755-72cf76f3b037_biasIsr_1123833_20.stderr parsl.dataflow.dflow DEBUG: Task 1325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1325 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1325 has memoization hash 4d5c928dad37c71d1cfca8e902876563 parsl.dataflow.memoization INFO: Task 1325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1325 try 0 launched on executor multi with executor id 1269 parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/692494e1-7df0-4d57-a859-a1d9acd6c469_biasIsr_1123837_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/692494e1-7df0-4d57-a859-a1d9acd6c469_biasIsr_1123837_20.stderr parsl.dataflow.dflow DEBUG: Task 1326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1326 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1326 has memoization hash 466ddc61e49409d9de7f8e0041e188a4 parsl.dataflow.memoization INFO: Task 1326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1326 try 0 launched on executor multi with executor id 1270 parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/61d9d2da-9966-45f7-9870-d241ea59388c_biasIsr_1123735_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/61d9d2da-9966-45f7-9870-d241ea59388c_biasIsr_1123735_20.stderr parsl.dataflow.dflow DEBUG: Task 1327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1327 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1327 has memoization hash d4682094363878b251ffa859b7c5d20c parsl.dataflow.memoization INFO: Task 1327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1327 try 0 launched on executor multi with executor id 1271 parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b7815653-eea7-4981-acce-ea5cbe6e808b_biasIsr_1123737_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/b7815653-eea7-4981-acce-ea5cbe6e808b_biasIsr_1123737_20.stderr parsl.dataflow.dflow DEBUG: Task 1328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1328 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1328 has memoization hash be011a639cf0ce79415a2a315e8c5492 parsl.dataflow.memoization INFO: Task 1328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1328 try 0 launched on executor multi with executor id 1272 parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e2b91b30-596f-4b22-b637-04cfadf916e4_biasIsr_1123835_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e2b91b30-596f-4b22-b637-04cfadf916e4_biasIsr_1123835_20.stderr parsl.dataflow.dflow DEBUG: Task 1329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1329 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1329 has memoization hash 6b8ff05901b15b5183355142dbe23aab parsl.dataflow.memoization INFO: Task 1329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1329 try 0 launched on executor multi with executor id 1273 parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/423d3759-d2e7-4d9b-8d08-6ead5e6da597_biasIsr_1123732_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/423d3759-d2e7-4d9b-8d08-6ead5e6da597_biasIsr_1123732_20.stderr parsl.dataflow.dflow DEBUG: Task 1330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1330 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1330 has memoization hash 4a3a228770036d7871b62de89a09fcb6 parsl.dataflow.memoization INFO: Task 1330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1330 try 0 launched on executor multi with executor id 1274 parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/ac32d1ba-b509-4eed-992c-867ea315c338_biasIsr_1123829_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/ac32d1ba-b509-4eed-992c-867ea315c338_biasIsr_1123829_20.stderr parsl.dataflow.dflow DEBUG: Task 1331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1331 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1331 has memoization hash 5a99cce75f54b1465249110b81428c39 parsl.dataflow.memoization INFO: Task 1331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1331 try 0 launched on executor multi with executor id 1275 parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6047f8cb-f60f-4930-b52b-27ca361e1ab1_biasIsr_1123834_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/6047f8cb-f60f-4930-b52b-27ca361e1ab1_biasIsr_1123834_20.stderr parsl.dataflow.dflow DEBUG: Task 1332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1332 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1332 has memoization hash 8450a285de38310b04cec8d3fc9da82b parsl.dataflow.memoization INFO: Task 1332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b51800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1332 try 0 launched on executor multi with executor id 1276 parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a22e839c-73e4-4a44-92e3-3be8e9430f64_biasIsr_1123738_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/a22e839c-73e4-4a44-92e3-3be8e9430f64_biasIsr_1123738_20.stderr parsl.dataflow.dflow DEBUG: Task 1333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1333 submitted for App cpBiasCombine, waiting on task 1311, task 1312, task 1313, task 1314, task 1315, task 1316, task 1317, task 1318, task 1319, task 1320, task 1321, task 1322, task 1323, task 1324, task 1325, task 1326, task 1327, task 1328, task 1329, task 1330, task 1331, task 1332 parsl.dataflow.dflow DEBUG: Task 1333 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1333 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1334 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1334 has memoization hash 84ddd53c2c9eac85a168d149138ef4e1 parsl.dataflow.memoization INFO: Task 1334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1334 try 0 launched on executor multi with executor id 1277 parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/7aaaeadc-0333-4d33-b1c9-a8fc2938c870_biasIsr_1123736_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/7aaaeadc-0333-4d33-b1c9-a8fc2938c870_biasIsr_1123736_44.stderr parsl.dataflow.dflow DEBUG: Task 1335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1335 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1335 has memoization hash 600d45ce19931515d4776007e986078e parsl.dataflow.memoization INFO: Task 1335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1335 try 0 launched on executor multi with executor id 1278 parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/47a4c91c-e21f-4603-8a7f-e8bed277322b_biasIsr_1123832_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/47a4c91c-e21f-4603-8a7f-e8bed277322b_biasIsr_1123832_44.stderr parsl.dataflow.dflow DEBUG: Task 1336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1336 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1336 has memoization hash 9a4825107d683a27eebec327f5bab9f1 parsl.dataflow.memoization INFO: Task 1336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1336 try 0 launched on executor multi with executor id 1279 parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1bc98444-e2c2-47d3-88cf-85c8ea4519c5_biasIsr_1123728_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/1bc98444-e2c2-47d3-88cf-85c8ea4519c5_biasIsr_1123728_44.stderr parsl.dataflow.dflow DEBUG: Task 1337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1337 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1337 has memoization hash de942d3e374486c132c299e088aa14be parsl.dataflow.memoization INFO: Task 1337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1337 try 0 launched on executor multi with executor id 1280 parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ba8a1a54-1c48-4406-9534-ac872c458d63_biasIsr_1123839_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/ba8a1a54-1c48-4406-9534-ac872c458d63_biasIsr_1123839_44.stderr parsl.dataflow.dflow DEBUG: Task 1338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1338 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1338 has memoization hash fe055e58ccadc0eb993e600bf40cef48 parsl.dataflow.memoization INFO: Task 1338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1338 try 0 launched on executor multi with executor id 1281 parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/cacc6ce7-060d-4d25-b30e-9ab0f049a796_biasIsr_1123731_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/cacc6ce7-060d-4d25-b30e-9ab0f049a796_biasIsr_1123731_44.stderr parsl.dataflow.dflow DEBUG: Task 1339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1339 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1339 has memoization hash aed4c003f9cf6876e29342c5800fecbf parsl.dataflow.memoization INFO: Task 1339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1339 try 0 launched on executor multi with executor id 1282 parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3d103cee-de3e-4546-a3e7-681bb292987e_biasIsr_1123834_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/3d103cee-de3e-4546-a3e7-681bb292987e_biasIsr_1123834_44.stderr parsl.dataflow.dflow DEBUG: Task 1340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1340 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1340 has memoization hash 3b5bd35f0925c08ea23f9086a3669edf parsl.dataflow.memoization INFO: Task 1340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1340 try 0 launched on executor multi with executor id 1283 parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/67ef33bc-881d-4d35-9cf0-fc53eda0a42d_biasIsr_1123838_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/67ef33bc-881d-4d35-9cf0-fc53eda0a42d_biasIsr_1123838_44.stderr parsl.dataflow.dflow DEBUG: Task 1341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1341 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1341 has memoization hash 5e2f21d2476a473db24a07ffa49db49f parsl.dataflow.memoization INFO: Task 1341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1341 try 0 launched on executor multi with executor id 1284 parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/5d291fc3-56b1-4f3c-b27f-dfa2b93d45e1_biasIsr_1123734_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/5d291fc3-56b1-4f3c-b27f-dfa2b93d45e1_biasIsr_1123734_44.stderr parsl.dataflow.dflow DEBUG: Task 1342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1342 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1342 has memoization hash 3d9ec2a8a8aa6b84b3b9c162ef6a2537 parsl.dataflow.memoization INFO: Task 1342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1342 try 0 launched on executor multi with executor id 1285 parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/56736de3-3992-40f6-ad8c-94533f0cc9fe_biasIsr_1123729_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/56736de3-3992-40f6-ad8c-94533f0cc9fe_biasIsr_1123729_44.stderr parsl.dataflow.dflow DEBUG: Task 1343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1343 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1343 has memoization hash 7f97db2f22189e771b3effd3dc48e3e5 parsl.dataflow.memoization INFO: Task 1343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1343 try 0 launched on executor multi with executor id 1286 parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0692b60c-e78d-40e5-b52c-7a7fbba66718_biasIsr_1123732_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/0692b60c-e78d-40e5-b52c-7a7fbba66718_biasIsr_1123732_44.stderr parsl.dataflow.dflow DEBUG: Task 1344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1344 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1344 has memoization hash cdf2e2f65780958e20dd020ffd0bca7d parsl.dataflow.memoization INFO: Task 1344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1344 try 0 launched on executor multi with executor id 1287 parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3e3ca542-0d39-43a1-b44b-363c3b82a0b3_biasIsr_1123831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/3e3ca542-0d39-43a1-b44b-363c3b82a0b3_biasIsr_1123831_44.stderr parsl.dataflow.dflow DEBUG: Task 1345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1345 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1345 has memoization hash 0069bf50c1bf8876c4c66a948b07f3d0 parsl.dataflow.memoization INFO: Task 1345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1345 try 0 launched on executor multi with executor id 1288 parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/227c448e-dd35-4488-ab08-4131f7dcf33b_biasIsr_1123837_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/227c448e-dd35-4488-ab08-4131f7dcf33b_biasIsr_1123837_44.stderr parsl.dataflow.dflow DEBUG: Task 1346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1346 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1346 has memoization hash f345c44b044431f648db1bb269c2b6fa parsl.dataflow.memoization INFO: Task 1346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1346 try 0 launched on executor multi with executor id 1289 parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f0bfb726-1f94-478d-9310-aedd14e6660b_biasIsr_1123735_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/f0bfb726-1f94-478d-9310-aedd14e6660b_biasIsr_1123735_44.stderr parsl.dataflow.dflow DEBUG: Task 1347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1347 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1347 has memoization hash 4f75592939d5e21ee01f5fd19aa7d2ec parsl.dataflow.memoization INFO: Task 1347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1347 try 0 launched on executor multi with executor id 1290 parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/0262ea0b-c6d0-4cac-a65b-c9d0ae583314_biasIsr_1123733_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/0262ea0b-c6d0-4cac-a65b-c9d0ae583314_biasIsr_1123733_44.stderr parsl.dataflow.dflow DEBUG: Task 1348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1348 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1348 has memoization hash 747655301b77e2eef84dd6d046fff855 parsl.dataflow.memoization INFO: Task 1348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1348 try 0 launched on executor multi with executor id 1291 parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/935ff4e3-9da6-48d7-bdda-22f45d5a13b7_biasIsr_1123833_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/935ff4e3-9da6-48d7-bdda-22f45d5a13b7_biasIsr_1123833_44.stderr parsl.dataflow.dflow DEBUG: Task 1349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1349 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1349 has memoization hash d4d0b86c83f6a2a48a09c68e2718fe4d parsl.dataflow.memoization INFO: Task 1349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b52fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1349 try 0 launched on executor multi with executor id 1292 parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/6a198c3a-cca9-40e8-99cf-c1a823dff433_biasIsr_1123730_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/6a198c3a-cca9-40e8-99cf-c1a823dff433_biasIsr_1123730_44.stderr parsl.dataflow.dflow DEBUG: Task 1350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1350 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1350 has memoization hash 8b4f3cc36d8b228758d2935bea15655c parsl.dataflow.memoization INFO: Task 1350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b53100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1350 try 0 launched on executor multi with executor id 1293 parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c969248b-02ec-44aa-801b-6be8f691002b_biasIsr_1123835_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/c969248b-02ec-44aa-801b-6be8f691002b_biasIsr_1123835_44.stderr parsl.dataflow.dflow DEBUG: Task 1351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1351 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1351 has memoization hash 1fe052db3a6bedbd19e5b0b347751477 parsl.dataflow.memoization INFO: Task 1351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b53060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1351 try 0 launched on executor multi with executor id 1294 parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e5e13e34-0236-4a04-ac4c-42f6df06c9f5_biasIsr_1123830_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/e5e13e34-0236-4a04-ac4c-42f6df06c9f5_biasIsr_1123830_44.stderr parsl.dataflow.dflow DEBUG: Task 1352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1352 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1352 has memoization hash 8f3bcffbf3d4d55e0b1a71efdc06441f parsl.dataflow.memoization INFO: Task 1352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b531a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1352 try 0 launched on executor multi with executor id 1295 parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cf17d762-301f-43a2-8936-62c86f4df416_biasIsr_1123737_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/cf17d762-301f-43a2-8936-62c86f4df416_biasIsr_1123737_44.stderr parsl.dataflow.dflow DEBUG: Task 1353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1353 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1353 has memoization hash da0415c6da753b8d717da16f82f89b14 parsl.dataflow.memoization INFO: Task 1353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b53240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1353 try 0 launched on executor multi with executor id 1296 parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/4eb4e92f-fcc1-4504-8f7b-a555380bf4ca_biasIsr_1123829_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/4eb4e92f-fcc1-4504-8f7b-a555380bf4ca_biasIsr_1123829_44.stderr parsl.dataflow.dflow DEBUG: Task 1354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1354 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1354 has memoization hash 324772127555bd51d32d86b5ee7eeec3 parsl.dataflow.memoization INFO: Task 1354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b532e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1354 try 0 launched on executor multi with executor id 1297 parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/deaf3f6e-00a8-4ccb-b03d-f839d2243fe0_biasIsr_1123738_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/deaf3f6e-00a8-4ccb-b03d-f839d2243fe0_biasIsr_1123738_44.stderr parsl.dataflow.dflow DEBUG: Task 1355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1355 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1355 has memoization hash 6141627da2e4cc7415f908cab7426149 parsl.dataflow.memoization INFO: Task 1355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9b53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1355 try 0 launched on executor multi with executor id 1298 parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8a1b41d6-ca18-4780-bebe-1f166a7a6e35_biasIsr_1123836_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/8a1b41d6-ca18-4780-bebe-1f166a7a6e35_biasIsr_1123836_44.stderr parsl.dataflow.dflow DEBUG: Task 1356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1356 submitted for App cpBiasCombine, waiting on task 1334, task 1335, task 1336, task 1337, task 1338, task 1339, task 1340, task 1341, task 1342, task 1343, task 1344, task 1345, task 1346, task 1347, task 1348, task 1349, task 1350, task 1351, task 1352, task 1353, task 1354, task 1355 parsl.dataflow.dflow DEBUG: Task 1356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1357 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1357 has memoization hash cfa337eff4ba8abec3cd0605ce2373b7 parsl.dataflow.memoization INFO: Task 1357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1357 try 0 launched on executor multi with executor id 1299 parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a51abfff-5d3b-49a0-8da7-1895031f4070_biasIsr_1123731_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/a51abfff-5d3b-49a0-8da7-1895031f4070_biasIsr_1123731_16.stderr parsl.dataflow.dflow DEBUG: Task 1358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1358 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1358 has memoization hash c476cef0e7c00bdef9bdedb148c427f9 parsl.dataflow.memoization INFO: Task 1358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1358 try 0 launched on executor multi with executor id 1300 parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b15606af-91ae-41f5-93aa-e514b78a160b_biasIsr_1123734_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/b15606af-91ae-41f5-93aa-e514b78a160b_biasIsr_1123734_16.stderr parsl.dataflow.dflow DEBUG: Task 1359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1359 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1359 has memoization hash 11561bbccc66e45f50d615e31f72ee97 parsl.dataflow.memoization INFO: Task 1359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1359 try 0 launched on executor multi with executor id 1301 parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/17986ca7-e439-47f9-b469-8cb5a4cdecfd_biasIsr_1123835_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/17986ca7-e439-47f9-b469-8cb5a4cdecfd_biasIsr_1123835_16.stderr parsl.dataflow.dflow DEBUG: Task 1360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1360 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1360 has memoization hash 7bbf66d4fc16f947c9e4cfd971b70f82 parsl.dataflow.memoization INFO: Task 1360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99985e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1360 try 0 launched on executor multi with executor id 1302 parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/07089553-45b8-4e75-86c2-1a04cd9ce5c1_biasIsr_1123830_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/07089553-45b8-4e75-86c2-1a04cd9ce5c1_biasIsr_1123830_16.stderr parsl.dataflow.dflow DEBUG: Task 1361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1361 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1361 has memoization hash 8546f3ccb1affb57350bce906f4b2f94 parsl.dataflow.memoization INFO: Task 1361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1361 try 0 launched on executor multi with executor id 1303 parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c64f1885-8548-4128-93fb-1035e113d10c_biasIsr_1123833_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/c64f1885-8548-4128-93fb-1035e113d10c_biasIsr_1123833_16.stderr parsl.dataflow.dflow DEBUG: Task 1362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1362 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1362 has memoization hash c81f169206bd8c08cf6a3271938e1c2c parsl.dataflow.memoization INFO: Task 1362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1362 try 0 launched on executor multi with executor id 1304 parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/281f8171-b45e-4bb6-87c9-355d46683b88_biasIsr_1123834_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/281f8171-b45e-4bb6-87c9-355d46683b88_biasIsr_1123834_16.stderr parsl.dataflow.dflow DEBUG: Task 1363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1363 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1363 has memoization hash aa8857b1f916976a9af533536b28ff89 parsl.dataflow.memoization INFO: Task 1363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1363 try 0 launched on executor multi with executor id 1305 parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/743ffbff-bf7e-44fc-a11f-25710b59578d_biasIsr_1123838_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/743ffbff-bf7e-44fc-a11f-25710b59578d_biasIsr_1123838_16.stderr parsl.dataflow.dflow DEBUG: Task 1364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1364 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1364 has memoization hash 7b15afa1d2499e7881e892aadfeec724 parsl.dataflow.memoization INFO: Task 1364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99987c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1364 try 0 launched on executor multi with executor id 1306 parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/18ed5296-53f8-4734-95e1-b951b2b1c5f4_biasIsr_1123732_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/18ed5296-53f8-4734-95e1-b951b2b1c5f4_biasIsr_1123732_16.stderr parsl.dataflow.dflow DEBUG: Task 1365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1365 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1365 has memoization hash a4d25edbbc1e9832e8fb1a5201c80cdf parsl.dataflow.memoization INFO: Task 1365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1365 try 0 launched on executor multi with executor id 1307 parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/64b2a09e-4182-4488-aad3-ce0374463b87_biasIsr_1123832_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/64b2a09e-4182-4488-aad3-ce0374463b87_biasIsr_1123832_16.stderr parsl.dataflow.dflow DEBUG: Task 1366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1366 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1366 has memoization hash 469b3d9ee6d7be577635c8692aa4cc08 parsl.dataflow.memoization INFO: Task 1366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1366 try 0 launched on executor multi with executor id 1308 parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/89603770-f448-410a-8829-e54d09bf8bc2_biasIsr_1123729_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/89603770-f448-410a-8829-e54d09bf8bc2_biasIsr_1123729_16.stderr parsl.dataflow.dflow DEBUG: Task 1367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1367 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1367 has memoization hash 777a365177bb92c4d4d3433f3fff8a21 parsl.dataflow.memoization INFO: Task 1367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1367 try 0 launched on executor multi with executor id 1309 parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2aa7f359-7bfd-45f3-a198-97e7b50ac9ef_biasIsr_1123837_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/2aa7f359-7bfd-45f3-a198-97e7b50ac9ef_biasIsr_1123837_16.stderr parsl.dataflow.dflow DEBUG: Task 1368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1368 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1368 has memoization hash eca9de173baf6e801c8512936c81f53b parsl.dataflow.memoization INFO: Task 1368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1368 try 0 launched on executor multi with executor id 1310 parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/acecd668-d7f8-41da-9268-85e3e26f6d7c_biasIsr_1123831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/acecd668-d7f8-41da-9268-85e3e26f6d7c_biasIsr_1123831_16.stderr parsl.dataflow.dflow DEBUG: Task 1369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1369 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1369 has memoization hash 82d0e227e8f6a727d42a409a98b65f48 parsl.dataflow.memoization INFO: Task 1369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1369 try 0 launched on executor multi with executor id 1311 parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e10f8e5b-8af0-424c-9c05-5a3ebf05786e_biasIsr_1123829_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/e10f8e5b-8af0-424c-9c05-5a3ebf05786e_biasIsr_1123829_16.stderr parsl.dataflow.dflow DEBUG: Task 1370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1370 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1370 has memoization hash 3fc315f65f5a844a547e04c0fcf6b94b parsl.dataflow.memoization INFO: Task 1370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1370 try 0 launched on executor multi with executor id 1312 parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/bba2e007-f7c4-4142-b403-69369ab68fb5_biasIsr_1123738_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/bba2e007-f7c4-4142-b403-69369ab68fb5_biasIsr_1123738_16.stderr parsl.dataflow.dflow DEBUG: Task 1371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1371 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1371 has memoization hash a488f8c4386bed8824330c086c037f10 parsl.dataflow.memoization INFO: Task 1371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1371 try 0 launched on executor multi with executor id 1313 parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d43d2efd-d0a0-46f9-8560-65bade620d75_biasIsr_1123735_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/d43d2efd-d0a0-46f9-8560-65bade620d75_biasIsr_1123735_16.stderr parsl.dataflow.dflow DEBUG: Task 1372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1372 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1372 has memoization hash 5a013eba23387e706842abaf136085b8 parsl.dataflow.memoization INFO: Task 1372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1372 try 0 launched on executor multi with executor id 1314 parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0d2b1707-392a-4e1d-bd91-9ff6f441453d_biasIsr_1123737_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/0d2b1707-392a-4e1d-bd91-9ff6f441453d_biasIsr_1123737_16.stderr parsl.dataflow.dflow DEBUG: Task 1373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1373 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1373 has memoization hash b08dd6c09a3c88782ba91977540d430f parsl.dataflow.memoization INFO: Task 1373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1373 try 0 launched on executor multi with executor id 1315 parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/fa2580a8-9226-449c-aca1-6fbb6174bbdd_biasIsr_1123730_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/fa2580a8-9226-449c-aca1-6fbb6174bbdd_biasIsr_1123730_16.stderr parsl.dataflow.dflow DEBUG: Task 1374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1374 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1374 has memoization hash 2127e325607c8f10741ee5d79fd78639 parsl.dataflow.memoization INFO: Task 1374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1374 try 0 launched on executor multi with executor id 1316 parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/1da525c4-cac8-44e7-ad51-501e45ad1999_biasIsr_1123733_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/1da525c4-cac8-44e7-ad51-501e45ad1999_biasIsr_1123733_16.stderr parsl.dataflow.dflow DEBUG: Task 1375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1375 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1375 has memoization hash d6ea87e8a776d2c3f33823be98212389 parsl.dataflow.memoization INFO: Task 1375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1375 try 0 launched on executor multi with executor id 1317 parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/f30aba52-6860-4973-bdc8-13757d3ddc57_biasIsr_1123728_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/f30aba52-6860-4973-bdc8-13757d3ddc57_biasIsr_1123728_16.stderr parsl.dataflow.dflow DEBUG: Task 1376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1376 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1376 has memoization hash 9554f88c4b69faeb7978d8cb153e7038 parsl.dataflow.memoization INFO: Task 1376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1376 try 0 launched on executor multi with executor id 1318 parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/56d4f7bf-3d48-4e9a-95ef-5052a73c7196_biasIsr_1123839_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/56d4f7bf-3d48-4e9a-95ef-5052a73c7196_biasIsr_1123839_16.stderr parsl.dataflow.dflow DEBUG: Task 1377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1377 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1377 has memoization hash e022dd887ba225f06b39666766219aca parsl.dataflow.memoization INFO: Task 1377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9998fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1377 try 0 launched on executor multi with executor id 1319 parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/eb13a0e1-063a-4a6d-86d3-4ffb242b3e6c_biasIsr_1123736_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/eb13a0e1-063a-4a6d-86d3-4ffb242b3e6c_biasIsr_1123736_16.stderr parsl.dataflow.dflow DEBUG: Task 1378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1378 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1378 has memoization hash 70be570e9d99c8a049e2430c3c9f42c2 parsl.dataflow.memoization INFO: Task 1378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9999080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1378 try 0 launched on executor multi with executor id 1320 parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/45b60afe-d111-4163-9d66-3db4b4e60462_biasIsr_1123836_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/45b60afe-d111-4163-9d66-3db4b4e60462_biasIsr_1123836_16.stderr parsl.dataflow.dflow DEBUG: Task 1379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1379 submitted for App cpBiasCombine, waiting on task 1357, task 1358, task 1359, task 1360, task 1361, task 1362, task 1363, task 1364, task 1365, task 1366, task 1367, task 1368, task 1369, task 1370, task 1371, task 1372, task 1373, task 1374, task 1375, task 1376, task 1377, task 1378 parsl.dataflow.dflow DEBUG: Task 1379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1380 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1380 has memoization hash 04b6d4605392527f140d6be9099f47ec parsl.dataflow.memoization INFO: Task 1380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9999f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1380 try 0 launched on executor multi with executor id 1321 parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/afdbcf7b-985d-42f9-806e-0b6c2b0d0257_biasIsr_1123734_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/afdbcf7b-985d-42f9-806e-0b6c2b0d0257_biasIsr_1123734_32.stderr parsl.dataflow.dflow DEBUG: Task 1381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1381 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1381 has memoization hash 173f674bf90a98e9694258d6974c5893 parsl.dataflow.memoization INFO: Task 1381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c9999ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1381 try 0 launched on executor multi with executor id 1322 parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/457891bd-daeb-47e2-8cfb-a961b8f515cc_biasIsr_1123732_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/457891bd-daeb-47e2-8cfb-a961b8f515cc_biasIsr_1123732_32.stderr parsl.dataflow.dflow DEBUG: Task 1382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1382 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1382 has memoization hash b2181102aa4e9815c92d5216696ad21a parsl.dataflow.memoization INFO: Task 1382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1382 try 0 launched on executor multi with executor id 1323 parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/feb3b43e-d2fd-440a-9d51-7fd5596d07ac_biasIsr_1123830_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/feb3b43e-d2fd-440a-9d51-7fd5596d07ac_biasIsr_1123830_32.stderr parsl.dataflow.dflow DEBUG: Task 1383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1383 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1383 has memoization hash 12f77f0dac13e432bafed0fec88b252d parsl.dataflow.memoization INFO: Task 1383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1383 try 0 launched on executor multi with executor id 1324 parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/964a71bd-9f4c-4034-b67c-c99242deff7e_biasIsr_1123733_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/964a71bd-9f4c-4034-b67c-c99242deff7e_biasIsr_1123733_32.stderr parsl.dataflow.dflow DEBUG: Task 1384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1384 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1384 has memoization hash d2ed288c02ee0cc16bdaaa400cc10acb parsl.dataflow.memoization INFO: Task 1384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1384 try 0 launched on executor multi with executor id 1325 parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/4fe81ee3-c0b2-433d-9374-b9adfb96ac18_biasIsr_1123737_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/4fe81ee3-c0b2-433d-9374-b9adfb96ac18_biasIsr_1123737_32.stderr parsl.dataflow.dflow DEBUG: Task 1385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1385 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1385 has memoization hash 584b11c7f16041acaf4f7303ee6ecfdb parsl.dataflow.memoization INFO: Task 1385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1385 try 0 launched on executor multi with executor id 1326 parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d1d019d9-4d64-49f3-ae3f-1498f00eafd1_biasIsr_1123839_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d1d019d9-4d64-49f3-ae3f-1498f00eafd1_biasIsr_1123839_32.stderr parsl.dataflow.dflow DEBUG: Task 1386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1386 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1386 has memoization hash 5d3f0f10711cb9a21f4d9f2252cab16b parsl.dataflow.memoization INFO: Task 1386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1386 try 0 launched on executor multi with executor id 1327 parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/04dd54d4-ab6f-48de-b595-bae0349c8337_biasIsr_1123829_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/04dd54d4-ab6f-48de-b595-bae0349c8337_biasIsr_1123829_32.stderr parsl.dataflow.dflow DEBUG: Task 1387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1387 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1387 has memoization hash ffe6095101c7c8ca0b4016fe9b33dd9c parsl.dataflow.memoization INFO: Task 1387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1387 try 0 launched on executor multi with executor id 1328 parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8409d1d1-fb8d-405c-8337-29a6c56a33f9_biasIsr_1123738_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/8409d1d1-fb8d-405c-8337-29a6c56a33f9_biasIsr_1123738_32.stderr parsl.dataflow.dflow DEBUG: Task 1388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1388 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1388 has memoization hash cc76992cb953e2654bf0baab3052f23e parsl.dataflow.memoization INFO: Task 1388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1388 try 0 launched on executor multi with executor id 1329 parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/adf424da-f12a-4961-9cfa-fb0394d5674f_biasIsr_1123831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/adf424da-f12a-4961-9cfa-fb0394d5674f_biasIsr_1123831_32.stderr parsl.dataflow.dflow DEBUG: Task 1389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1389 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1389 has memoization hash 835c22bec80f8f02c9902b727d0421bf parsl.dataflow.memoization INFO: Task 1389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1389 try 0 launched on executor multi with executor id 1330 parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a0f355f5-e4fd-4fe9-8a4c-305af04a1766_biasIsr_1123836_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a0f355f5-e4fd-4fe9-8a4c-305af04a1766_biasIsr_1123836_32.stderr parsl.dataflow.dflow DEBUG: Task 1390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1390 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1390 has memoization hash 595a74faea7e0af0c3233ed13c160826 parsl.dataflow.memoization INFO: Task 1390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1390 try 0 launched on executor multi with executor id 1331 parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ba76274b-3bdf-4e9a-b5cc-87605e5d1384_biasIsr_1123729_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/ba76274b-3bdf-4e9a-b5cc-87605e5d1384_biasIsr_1123729_32.stderr parsl.dataflow.dflow DEBUG: Task 1391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1391 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1391 has memoization hash 18f1b98fa9781e600bb7bbcb85b09cfc parsl.dataflow.memoization INFO: Task 1391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1391 try 0 launched on executor multi with executor id 1332 parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8cd22d57-6531-4a05-b6ac-0714a796ff23_biasIsr_1123728_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/8cd22d57-6531-4a05-b6ac-0714a796ff23_biasIsr_1123728_32.stderr parsl.dataflow.dflow DEBUG: Task 1392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1392 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1392 has memoization hash 484ea04147735718bb9f82cfe18724ce parsl.dataflow.memoization INFO: Task 1392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1392 try 0 launched on executor multi with executor id 1333 parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/c077d332-0f00-4ee1-a8f8-4c945ba1bf6e_biasIsr_1123736_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/c077d332-0f00-4ee1-a8f8-4c945ba1bf6e_biasIsr_1123736_32.stderr parsl.dataflow.dflow DEBUG: Task 1393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1393 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1393 has memoization hash 564244f5dfcc8b8d6870366f363ef4f1 parsl.dataflow.memoization INFO: Task 1393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1393 try 0 launched on executor multi with executor id 1334 parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/fb73d253-c8df-4f67-909d-1ac077147c3d_biasIsr_1123735_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/fb73d253-c8df-4f67-909d-1ac077147c3d_biasIsr_1123735_32.stderr parsl.dataflow.dflow DEBUG: Task 1394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1394 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1394 has memoization hash 5a9851b9ac85d904735eedbb72e64a74 parsl.dataflow.memoization INFO: Task 1394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1394 try 0 launched on executor multi with executor id 1335 parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c2942b3d-dcb6-41a6-bf2b-7b93dc1210cf_biasIsr_1123832_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/c2942b3d-dcb6-41a6-bf2b-7b93dc1210cf_biasIsr_1123832_32.stderr parsl.dataflow.dflow DEBUG: Task 1395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1395 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1395 has memoization hash f305d78acfe0968f7cedf29e993afcad parsl.dataflow.memoization INFO: Task 1395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1395 try 0 launched on executor multi with executor id 1336 parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/417700e8-af2e-438b-a26a-f84e96f735f5_biasIsr_1123731_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/417700e8-af2e-438b-a26a-f84e96f735f5_biasIsr_1123731_32.stderr parsl.dataflow.dflow DEBUG: Task 1396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1396 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1396 has memoization hash e8be84ca9e88fbeba2e9f877dd7ae948 parsl.dataflow.memoization INFO: Task 1396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1396 try 0 launched on executor multi with executor id 1337 parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/31b7029f-da0b-4827-a87e-ccb53c5cd8db_biasIsr_1123834_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/31b7029f-da0b-4827-a87e-ccb53c5cd8db_biasIsr_1123834_32.stderr parsl.dataflow.dflow DEBUG: Task 1397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1397 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1397 has memoization hash 3cee6a2fd7fb69af4f6a5ad695f94a27 parsl.dataflow.memoization INFO: Task 1397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1397 try 0 launched on executor multi with executor id 1338 parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/43269ea9-b2d8-460b-b5c4-59515addf1e3_biasIsr_1123833_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/43269ea9-b2d8-460b-b5c4-59515addf1e3_biasIsr_1123833_32.stderr parsl.dataflow.dflow DEBUG: Task 1398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1398 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1398 has memoization hash 5465df1f7edd96cdc3560e33c2daf986 parsl.dataflow.memoization INFO: Task 1398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1398 try 0 launched on executor multi with executor id 1339 parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e2394f6a-bfe0-416c-98b8-853eebbcb449_biasIsr_1123835_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/e2394f6a-bfe0-416c-98b8-853eebbcb449_biasIsr_1123835_32.stderr parsl.dataflow.dflow DEBUG: Task 1399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1399 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1399 has memoization hash 1994ed6b12771d4c1739a60ef9607860 parsl.dataflow.memoization INFO: Task 1399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1399 try 0 launched on executor multi with executor id 1340 parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a6faffd1-52d5-47a4-b121-45e587a0b949_biasIsr_1123730_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/a6faffd1-52d5-47a4-b121-45e587a0b949_biasIsr_1123730_32.stderr parsl.dataflow.dflow DEBUG: Task 1400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1400 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1400 has memoization hash a4ce36176a7e9d9dc0bf26857a92b022 parsl.dataflow.memoization INFO: Task 1400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1400 try 0 launched on executor multi with executor id 1341 parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/8032a06d-2d5b-4aa7-9397-b340f0ed6b8a_biasIsr_1123837_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/8032a06d-2d5b-4aa7-9397-b340f0ed6b8a_biasIsr_1123837_32.stderr parsl.dataflow.dflow DEBUG: Task 1401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1401 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1401 has memoization hash c57f975e67777244eb7f05de2d006d1f parsl.dataflow.memoization INFO: Task 1401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1401 try 0 launched on executor multi with executor id 1342 parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/edefb508-bbac-48bd-bcf6-249e9f59a8a2_biasIsr_1123838_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/edefb508-bbac-48bd-bcf6-249e9f59a8a2_biasIsr_1123838_32.stderr parsl.dataflow.dflow DEBUG: Task 1402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1402 submitted for App cpBiasCombine, waiting on task 1380, task 1381, task 1382, task 1383, task 1384, task 1385, task 1386, task 1387, task 1388, task 1389, task 1390, task 1391, task 1392, task 1393, task 1394, task 1395, task 1396, task 1397, task 1398, task 1399, task 1400, task 1401 parsl.dataflow.dflow DEBUG: Task 1402 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1402 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1403 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1403 has memoization hash c861ed005406dea0dce31e4dd2535ada parsl.dataflow.memoization INFO: Task 1403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1403 try 0 launched on executor multi with executor id 1343 parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/8736aa74-4115-4777-8dfc-b837cdc3a4b3_biasIsr_1123833_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/8736aa74-4115-4777-8dfc-b837cdc3a4b3_biasIsr_1123833_13.stderr parsl.dataflow.dflow DEBUG: Task 1404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1404 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1404 has memoization hash 5cbdebad85ac24530e60aeab7156aa0b parsl.dataflow.memoization INFO: Task 1404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1404 try 0 launched on executor multi with executor id 1344 parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ef34903f-0347-4709-b9a6-1a35edd22e83_biasIsr_1123838_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/ef34903f-0347-4709-b9a6-1a35edd22e83_biasIsr_1123838_13.stderr parsl.dataflow.dflow DEBUG: Task 1405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1405 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1405 has memoization hash 1766e2d188da5b2e1c99f7474f5a36fc parsl.dataflow.memoization INFO: Task 1405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1405 try 0 launched on executor multi with executor id 1345 parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/20e62d56-5a2b-4f16-9d4e-074545384edf_biasIsr_1123832_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/20e62d56-5a2b-4f16-9d4e-074545384edf_biasIsr_1123832_13.stderr parsl.dataflow.dflow DEBUG: Task 1406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1406 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1406 has memoization hash 8061e51986c74ca943f1bdc34454bcbc parsl.dataflow.memoization INFO: Task 1406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1406 try 0 launched on executor multi with executor id 1346 parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/70720410-6f96-4242-b02e-a1c521098d3f_biasIsr_1123829_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/70720410-6f96-4242-b02e-a1c521098d3f_biasIsr_1123829_13.stderr parsl.dataflow.dflow DEBUG: Task 1407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1407 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1407 has memoization hash 49a18bf77cee3f4ad0bedc1e06925e29 parsl.dataflow.memoization INFO: Task 1407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1407 try 0 launched on executor multi with executor id 1347 parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/fac6d38c-7d17-49d1-9775-3de3b6729a99_biasIsr_1123835_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/fac6d38c-7d17-49d1-9775-3de3b6729a99_biasIsr_1123835_13.stderr parsl.dataflow.dflow DEBUG: Task 1408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1408 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1408 has memoization hash 66fd529b591400bb37175e8228f2810b parsl.dataflow.memoization INFO: Task 1408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1408 try 0 launched on executor multi with executor id 1348 parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/6479e394-cf56-418e-b3cd-aacf41380dc0_biasIsr_1123839_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/6479e394-cf56-418e-b3cd-aacf41380dc0_biasIsr_1123839_13.stderr parsl.dataflow.dflow DEBUG: Task 1409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1409 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1409 has memoization hash 76bd60322ffa7a379f404a77b0e4d100 parsl.dataflow.memoization INFO: Task 1409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1409 try 0 launched on executor multi with executor id 1349 parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/e0f7470a-08ae-4894-842c-5328c4b678a9_biasIsr_1123736_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/e0f7470a-08ae-4894-842c-5328c4b678a9_biasIsr_1123736_13.stderr parsl.dataflow.dflow DEBUG: Task 1410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1410 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1410 has memoization hash 839bb91bf7b23baeada865c5287ec024 parsl.dataflow.memoization INFO: Task 1410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c999bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1410 try 0 launched on executor multi with executor id 1350 parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/61bdb13e-7223-4096-a67f-941e8e079714_biasIsr_1123837_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/61bdb13e-7223-4096-a67f-941e8e079714_biasIsr_1123837_13.stderr parsl.dataflow.dflow DEBUG: Task 1411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1411 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1411 has memoization hash b9f1993e5cb82a4b31ed6532f30773b9 parsl.dataflow.memoization INFO: Task 1411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1411 try 0 launched on executor multi with executor id 1351 parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/493acae6-5ee4-4df8-9bd0-2397d6bdda71_biasIsr_1123730_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/493acae6-5ee4-4df8-9bd0-2397d6bdda71_biasIsr_1123730_13.stderr parsl.dataflow.dflow DEBUG: Task 1412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1412 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1412 has memoization hash 1e5eaf3912f3aa41d2f774b0b9f2cce7 parsl.dataflow.memoization INFO: Task 1412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1412 try 0 launched on executor multi with executor id 1352 parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d39f6b03-e729-44aa-b309-33241f88edc3_biasIsr_1123728_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/d39f6b03-e729-44aa-b309-33241f88edc3_biasIsr_1123728_13.stderr parsl.dataflow.dflow DEBUG: Task 1413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1413 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1413 has memoization hash 837facf4b10fbd71ea35ce598e74a3d3 parsl.dataflow.memoization INFO: Task 1413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1413 try 0 launched on executor multi with executor id 1353 parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/44de724f-1820-4df2-a33d-816c0c808847_biasIsr_1123734_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/44de724f-1820-4df2-a33d-816c0c808847_biasIsr_1123734_13.stderr parsl.dataflow.dflow DEBUG: Task 1414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1414 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1414 has memoization hash 7cae5f42f3e7104070e632ca9549e709 parsl.dataflow.memoization INFO: Task 1414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1414 try 0 launched on executor multi with executor id 1354 parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/436a406b-b3c0-4d42-a6f0-9de3eb2f30e4_biasIsr_1123830_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/436a406b-b3c0-4d42-a6f0-9de3eb2f30e4_biasIsr_1123830_13.stderr parsl.dataflow.dflow DEBUG: Task 1415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1415 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1415 has memoization hash cae0917f7dc2aaad3cfe88347d612c67 parsl.dataflow.memoization INFO: Task 1415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1415 try 0 launched on executor multi with executor id 1355 parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/06fb0676-cd5f-4282-a3eb-f35988a02f5a_biasIsr_1123735_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/06fb0676-cd5f-4282-a3eb-f35988a02f5a_biasIsr_1123735_13.stderr parsl.dataflow.dflow DEBUG: Task 1416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1416 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1416 has memoization hash 3dc93cc14649bdf9d625717c255065b4 parsl.dataflow.memoization INFO: Task 1416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1416 try 0 launched on executor multi with executor id 1356 parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/47cc0fce-9705-4724-a448-f145341e6c45_biasIsr_1123733_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/47cc0fce-9705-4724-a448-f145341e6c45_biasIsr_1123733_13.stderr parsl.dataflow.dflow DEBUG: Task 1417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1417 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1417 has memoization hash 862d4a165e9a1c35588462da7f05f2f2 parsl.dataflow.memoization INFO: Task 1417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1417 try 0 launched on executor multi with executor id 1357 parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/52f613c8-e295-42ae-b985-55f4b886ceb0_biasIsr_1123729_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/52f613c8-e295-42ae-b985-55f4b886ceb0_biasIsr_1123729_13.stderr parsl.dataflow.dflow DEBUG: Task 1418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1418 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1418 has memoization hash 0002948c918b9b7e87146e140b47a293 parsl.dataflow.memoization INFO: Task 1418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1418 try 0 launched on executor multi with executor id 1358 parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/812a7ed9-9518-401a-8022-e68eb3ea1ef3_biasIsr_1123737_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/812a7ed9-9518-401a-8022-e68eb3ea1ef3_biasIsr_1123737_13.stderr parsl.dataflow.dflow DEBUG: Task 1419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1419 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1419 has memoization hash 0ba030d2f4c8443492788bcdbfcf12e5 parsl.dataflow.memoization INFO: Task 1419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1419 try 0 launched on executor multi with executor id 1359 parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b38923d9-7ebe-4ccb-a557-8a32204b0ce2_biasIsr_1123831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/b38923d9-7ebe-4ccb-a557-8a32204b0ce2_biasIsr_1123831_13.stderr parsl.dataflow.dflow DEBUG: Task 1420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1420 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1420 has memoization hash ddcb9220da510e789d1a5e5359af28de parsl.dataflow.memoization INFO: Task 1420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1420 try 0 launched on executor multi with executor id 1360 parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/fcc1949e-bafa-4c55-8ede-f18423881117_biasIsr_1123731_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/fcc1949e-bafa-4c55-8ede-f18423881117_biasIsr_1123731_13.stderr parsl.dataflow.dflow DEBUG: Task 1421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1421 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1421 has memoization hash 3c1fa41b8da293a91b62b5f213515441 parsl.dataflow.memoization INFO: Task 1421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1421 try 0 launched on executor multi with executor id 1361 parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/48ce6be5-8e0b-430a-8762-4e565856d006_biasIsr_1123738_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/48ce6be5-8e0b-430a-8762-4e565856d006_biasIsr_1123738_13.stderr parsl.dataflow.dflow DEBUG: Task 1422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1422 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1422 has memoization hash 3d63e38b61dd53c56184e51d6b721b3f parsl.dataflow.memoization INFO: Task 1422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1422 try 0 launched on executor multi with executor id 1362 parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/566c4020-adb1-48f4-86b2-e7c991f6c348_biasIsr_1123834_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/566c4020-adb1-48f4-86b2-e7c991f6c348_biasIsr_1123834_13.stderr parsl.dataflow.dflow DEBUG: Task 1423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1423 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1423 has memoization hash 76e6d1c728fcf35dbf3c4945997b3254 parsl.dataflow.memoization INFO: Task 1423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1423 try 0 launched on executor multi with executor id 1363 parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/29e8fed2-cd14-47ff-b394-6e9ae050e88d_biasIsr_1123732_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/29e8fed2-cd14-47ff-b394-6e9ae050e88d_biasIsr_1123732_13.stderr parsl.dataflow.dflow DEBUG: Task 1424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1424 submitted for App biasIsr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1424 has memoization hash 5c8afc2f3ad10c5864bd192d11b0596e parsl.dataflow.memoization INFO: Task 1424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502c99e4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1424 try 0 launched on executor multi with executor id 1364 parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2de74f8b-dd05-47fc-acdd-a668fdf2c99b_biasIsr_1123836_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/2de74f8b-dd05-47fc-acdd-a668fdf2c99b_biasIsr_1123836_13.stderr parsl.dataflow.dflow DEBUG: Task 1425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1425 submitted for App cpBiasCombine, waiting on task 1403, task 1404, task 1405, task 1406, task 1407, task 1408, task 1409, task 1410, task 1411, task 1412, task 1413, task 1414, task 1415, task 1416, task 1417, task 1418, task 1419, task 1420, task 1421, task 1422, task 1423, task 1424 parsl.dataflow.dflow DEBUG: Task 1425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1364 parsl.dataflow.strategy DEBUG: Executor multi has 1364 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-214.1709793867.6131566.submit parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-215.1709793867.6137087.submit parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-216.1709793867.6141653.submit parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-217.1709793867.6146057.submit parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-218.1709793867.6150486.submit parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-219.1709793867.6154778.submit parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-220.1709793867.6159117.submit parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-221.1709793867.6163428.submit parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-222.1709793867.616763.submit parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-223.1709793867.6171992.submit parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-224.1709793867.6176274.submit parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-225.1709793867.6180563.submit parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-226.1709793867.6184742.submit parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-227.1709793867.6189027.submit parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-228.1709793867.6193237.submit parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-229.1709793867.619749.submit parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-230.1709793867.620179.submit parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-231.1709793867.620602.submit parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-232.1709793867.621027.submit parsl.executors.status_handling INFO: Allocated block ID 233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-233 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-233.1709793867.6214466.submit parsl.executors.status_handling INFO: Allocated block ID 234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-234 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-234.1709793867.6218784.submit parsl.executors.status_handling INFO: Allocated block ID 235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-235 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-235.1709793867.6222997.submit parsl.executors.status_handling INFO: Allocated block ID 236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-236 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-236.1709793867.6227248.submit parsl.executors.status_handling INFO: Allocated block ID 237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-237 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-237.1709793867.6231604.submit parsl.executors.status_handling INFO: Allocated block ID 238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-238 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-238.1709793867.6235776.submit parsl.executors.status_handling INFO: Allocated block ID 239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-239 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-239.1709793867.6240008.submit parsl.executors.status_handling INFO: Allocated block ID 240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-240 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-240.1709793867.624419.submit parsl.executors.status_handling INFO: Allocated block ID 241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-241 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-241.1709793867.6379595.submit parsl.executors.status_handling INFO: Allocated block ID 242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-242 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-242.1709793867.6384766.submit parsl.executors.status_handling INFO: Allocated block ID 243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-243 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-243.1709793867.638917.submit parsl.executors.status_handling INFO: Allocated block ID 244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-244 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-244.1709793867.6393452.submit parsl.executors.status_handling INFO: Allocated block ID 245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-245 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-245.1709793867.639768.submit parsl.executors.status_handling INFO: Allocated block ID 246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-246 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-246.1709793867.6401963.submit parsl.executors.status_handling INFO: Allocated block ID 247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-247 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-247.1709793867.6406121.submit parsl.executors.status_handling INFO: Allocated block ID 248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-248 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-248.1709793867.6410353.submit parsl.executors.status_handling INFO: Allocated block ID 249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-249 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-249.1709793867.641452.submit parsl.executors.status_handling INFO: Allocated block ID 250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-250 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-250.1709793867.6418881.submit parsl.executors.status_handling INFO: Allocated block ID 251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-251 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-251.1709793867.642326.submit parsl.executors.status_handling INFO: Allocated block ID 252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-252 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-252.1709793867.6427653.submit parsl.executors.status_handling INFO: Allocated block ID 253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-253 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-253.1709793867.6432226.submit parsl.executors.status_handling INFO: Allocated block ID 254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-254 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-254.1709793867.643644.submit parsl.executors.status_handling INFO: Allocated block ID 255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-255 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-255.1709793867.6440716.submit parsl.executors.status_handling INFO: Allocated block ID 256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-256 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-256.1709793867.6444957.submit parsl.executors.status_handling INFO: Allocated block ID 257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-257 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-257.1709793867.644936.submit parsl.executors.status_handling INFO: Allocated block ID 258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-258 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-258.1709793867.6453648.submit parsl.executors.status_handling INFO: Allocated block ID 259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-259 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-259.1709793867.645793.submit parsl.executors.status_handling INFO: Allocated block ID 260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-260 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-260.1709793867.6462235.submit parsl.executors.status_handling INFO: Allocated block ID 261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-261 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-261.1709793867.6466439.submit parsl.executors.status_handling INFO: Allocated block ID 262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-262 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-262.1709793867.6470728.submit parsl.executors.status_handling INFO: Allocated block ID 263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-263 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-263.1709793867.6474974.submit parsl.executors.status_handling INFO: Allocated block ID 264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-264 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-264.1709793867.6479223.submit parsl.executors.status_handling INFO: Allocated block ID 265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-265 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-265.1709793867.6483452.submit parsl.executors.status_handling INFO: Allocated block ID 266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-266 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-266.1709793867.6487615.submit parsl.executors.status_handling INFO: Allocated block ID 267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-267 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-267.1709793867.6491957.submit parsl.executors.status_handling INFO: Allocated block ID 268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-268 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-268.1709793867.6496162.submit parsl.executors.status_handling INFO: Allocated block ID 269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-269 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-269.1709793867.6500428.submit parsl.executors.status_handling INFO: Allocated block ID 270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-270 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-270.1709793867.6504617.submit parsl.executors.status_handling INFO: Allocated block ID 271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-271 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-271.1709793867.6570125.submit parsl.executors.status_handling INFO: Allocated block ID 272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-272 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-272.1709793867.6575246.submit parsl.executors.status_handling INFO: Allocated block ID 273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-273 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-273.1709793867.6580148.submit parsl.executors.status_handling INFO: Allocated block ID 274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-274 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-274.1709793867.6584806.submit parsl.executors.status_handling INFO: Allocated block ID 275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-275 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-275.1709793867.6589694.submit parsl.executors.status_handling INFO: Allocated block ID 276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-276 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-276.1709793867.6594424.submit parsl.executors.status_handling INFO: Allocated block ID 277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-277 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-277.1709793867.6599126.submit parsl.executors.status_handling INFO: Allocated block ID 278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-278 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-278.1709793867.6603632.submit parsl.executors.status_handling INFO: Allocated block ID 279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-279 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-279.1709793867.6607854.submit parsl.executors.status_handling INFO: Allocated block ID 280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-280 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-280.1709793867.6612117.submit parsl.executors.status_handling INFO: Allocated block ID 281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-281 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-281.1709793867.6616318.submit parsl.executors.status_handling INFO: Allocated block ID 282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-282 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-282.1709793867.66206.submit parsl.executors.status_handling INFO: Allocated block ID 283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-283 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-283.1709793867.6624842.submit parsl.executors.status_handling INFO: Allocated block ID 284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-284 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-284.1709793867.66291.submit parsl.executors.status_handling INFO: Allocated block ID 285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-285 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-285.1709793867.6633282.submit parsl.executors.status_handling INFO: Allocated block ID 286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-286 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-286.1709793867.6637492.submit parsl.executors.status_handling INFO: Allocated block ID 287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-287 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-287.1709793867.6641805.submit parsl.executors.status_handling INFO: Allocated block ID 288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-288 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-288.1709793867.6646016.submit parsl.executors.status_handling INFO: Allocated block ID 289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-289 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-289.1709793867.665037.submit parsl.executors.status_handling INFO: Allocated block ID 290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-290 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-290.1709793867.6654625.submit parsl.executors.status_handling INFO: Allocated block ID 291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-291 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-291.1709793867.6658895.submit parsl.executors.status_handling INFO: Allocated block ID 292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-292 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-292.1709793867.6663105.submit parsl.executors.status_handling INFO: Allocated block ID 293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-293 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-293.1709793867.666733.submit parsl.executors.status_handling INFO: Allocated block ID 294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-294 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-294.1709793867.6671667.submit parsl.executors.status_handling INFO: Allocated block ID 295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-295 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-295.1709793867.6675847.submit parsl.executors.status_handling INFO: Allocated block ID 296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-296 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-296.1709793867.6684978.submit parsl.executors.status_handling INFO: Allocated block ID 297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-297 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-297.1709793867.6690006.submit parsl.executors.status_handling INFO: Allocated block ID 298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-298 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-298.1709793867.6694455.submit parsl.executors.status_handling INFO: Allocated block ID 299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-299 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-299.1709793867.6699.submit parsl.executors.status_handling INFO: Allocated block ID 300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-300 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-300.1709793867.670328.submit parsl.executors.status_handling INFO: Allocated block ID 301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-301 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-301.1709793867.6707587.submit parsl.executors.status_handling INFO: Allocated block ID 302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-302 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-302.1709793867.6711926.submit parsl.executors.status_handling INFO: Allocated block ID 303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-303 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-303.1709793867.6716158.submit parsl.executors.status_handling INFO: Allocated block ID 304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-304 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-304.1709793867.6720405.submit parsl.executors.status_handling INFO: Allocated block ID 305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-305 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-305.1709793867.6724617.submit parsl.executors.status_handling INFO: Allocated block ID 306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-306 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-306.1709793867.6728873.submit parsl.executors.status_handling INFO: Allocated block ID 307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-307 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-307.1709793867.6733048.submit parsl.executors.status_handling INFO: Allocated block ID 308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-308 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-308.1709793867.6737256.submit parsl.executors.status_handling INFO: Allocated block ID 309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-309 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-309.1709793867.674241.submit parsl.executors.status_handling INFO: Allocated block ID 310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-310 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-310.1709793867.6746953.submit parsl.executors.status_handling INFO: Allocated block ID 311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-311 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-311.1709793867.6751447.submit parsl.executors.status_handling INFO: Allocated block ID 312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-312 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-312 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-312.1709793867.6755862.submit parsl.executors.status_handling INFO: Allocated block ID 313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-313 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-313 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-313.1709793867.6760473.submit parsl.executors.status_handling INFO: Allocated block ID 314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-314 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-314 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-314.1709793867.6764908.submit parsl.executors.status_handling INFO: Allocated block ID 315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-315 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-315 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-315.1709793867.6769633.submit parsl.executors.status_handling INFO: Allocated block ID 316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-316 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-316 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-316.1709793867.6773925.submit parsl.executors.status_handling INFO: Allocated block ID 317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-317 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-317 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-317.1709793867.6778214.submit parsl.executors.status_handling INFO: Allocated block ID 318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-318 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-318 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-318.1709793867.6782558.submit parsl.executors.status_handling INFO: Allocated block ID 319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-319 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-319 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-319.1709793867.6786802.submit parsl.executors.status_handling INFO: Allocated block ID 320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-320 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-320 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-320.1709793867.6791077.submit parsl.executors.status_handling INFO: Allocated block ID 321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-321 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-321 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-321.1709793867.679535.submit parsl.executors.status_handling INFO: Allocated block ID 322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-322 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-322 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-322.1709793867.679965.submit parsl.executors.status_handling INFO: Allocated block ID 323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-323 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-323 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-323.1709793867.6803918.submit parsl.executors.status_handling INFO: Allocated block ID 324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-324 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-324 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-324.1709793867.6808236.submit parsl.executors.status_handling INFO: Allocated block ID 325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-325 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-325 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-325.1709793867.6812541.submit parsl.executors.status_handling INFO: Allocated block ID 326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-326 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-326 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-326.1709793867.681681.submit parsl.executors.status_handling INFO: Allocated block ID 327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-327 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-327 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-327.1709793867.6821218.submit parsl.executors.status_handling INFO: Allocated block ID 328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-328 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-328 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-328.1709793867.6825485.submit parsl.executors.status_handling INFO: Allocated block ID 329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-329 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-329 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-329.1709793867.682984.submit parsl.executors.status_handling INFO: Allocated block ID 330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-330 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-330 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-330.1709793867.683413.submit parsl.executors.status_handling INFO: Allocated block ID 331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-331 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-331 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-331.1709793867.6838517.submit parsl.executors.status_handling INFO: Allocated block ID 332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-332 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-332 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-332.1709793867.6842809.submit parsl.executors.status_handling INFO: Allocated block ID 333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-333 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-333 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-333.1709793867.6847034.submit parsl.executors.status_handling INFO: Allocated block ID 334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-334 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-334 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-334.1709793867.6853998.submit parsl.executors.status_handling INFO: Allocated block ID 335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-335 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-335 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-335.1709793867.685863.submit parsl.executors.status_handling INFO: Allocated block ID 336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-336 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-336 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-336.1709793867.6862984.submit parsl.executors.status_handling INFO: Allocated block ID 337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-337 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-337 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-337.1709793867.6867461.submit parsl.executors.status_handling INFO: Allocated block ID 338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-338 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-338 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-338.1709793867.687179.submit parsl.executors.status_handling INFO: Allocated block ID 339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-339 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-339 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-339.1709793867.687601.submit parsl.executors.status_handling INFO: Allocated block ID 340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-340 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-340 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-340.1709793867.6880298.submit parsl.executors.status_handling INFO: Allocated block ID 341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-341 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-341 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-341.1709793867.6884577.submit parsl.executors.status_handling INFO: Allocated block ID 342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-342 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-342 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-342.1709793867.6888921.submit parsl.executors.status_handling INFO: Allocated block ID 343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-343 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-343 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-343.1709793867.6893582.submit parsl.executors.status_handling INFO: Allocated block ID 344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-344 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-344 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-344.1709793867.6897883.submit parsl.executors.status_handling INFO: Allocated block ID 345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-345 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-345 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-345.1709793867.6902168.submit parsl.executors.status_handling INFO: Allocated block ID 346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-346 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-346 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-346.1709793867.6906383.submit parsl.executors.status_handling INFO: Allocated block ID 347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-347 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-347 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-347.1709793867.6910703.submit parsl.executors.status_handling INFO: Allocated block ID 348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-348 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-348 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-348.1709793867.6914892.submit parsl.executors.status_handling INFO: Allocated block ID 349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-349 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-349 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-349.1709793867.6919246.submit parsl.executors.status_handling INFO: Allocated block ID 350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-350 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-350 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-350.1709793867.6923466.submit parsl.executors.status_handling INFO: Allocated block ID 351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-351 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-351 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-351.1709793867.6927705.submit parsl.executors.status_handling INFO: Allocated block ID 352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-352 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-352 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-352.1709793867.6931975.submit parsl.executors.status_handling INFO: Allocated block ID 353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-353 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-353 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-353.1709793867.6936214.submit parsl.executors.status_handling INFO: Allocated block ID 354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-354 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-354 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-354.1709793867.6940491.submit parsl.executors.status_handling INFO: Allocated block ID 355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-355 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-355 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-355.1709793867.6944704.submit parsl.executors.status_handling INFO: Allocated block ID 356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-356 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-356 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-356.1709793867.694897.submit parsl.executors.status_handling INFO: Allocated block ID 357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-357 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-357 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-357.1709793867.695331.submit parsl.executors.status_handling INFO: Allocated block ID 358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-358 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-358 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-358.1709793867.6957538.submit parsl.executors.status_handling INFO: Allocated block ID 359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-359 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-359 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-359.1709793867.6961913.submit parsl.executors.status_handling INFO: Allocated block ID 360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-360 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-360 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-360.1709793867.6966095.submit parsl.executors.status_handling INFO: Allocated block ID 361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-361 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-361 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-361.1709793867.6970396.submit parsl.executors.status_handling INFO: Allocated block ID 362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-362 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-362 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-362.1709793867.6974635.submit parsl.executors.status_handling INFO: Allocated block ID 363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-363 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-363 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-363.1709793867.6978972.submit parsl.executors.status_handling INFO: Allocated block ID 364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-364 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-364 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-364.1709793867.6983192.submit parsl.executors.status_handling INFO: Allocated block ID 365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-365 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-365 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-365.1709793867.6987443.submit parsl.executors.status_handling INFO: Allocated block ID 366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-366 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-366 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-366.1709793867.699174.submit parsl.executors.status_handling INFO: Allocated block ID 367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-367 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-367 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-367.1709793867.6995916.submit parsl.executors.status_handling INFO: Allocated block ID 368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-368 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-368 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-368.1709793867.7000213.submit parsl.executors.status_handling INFO: Allocated block ID 369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-369 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-369 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-369.1709793867.7004406.submit parsl.executors.status_handling INFO: Allocated block ID 370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-370 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-370 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-370.1709793867.700865.submit parsl.executors.status_handling INFO: Allocated block ID 371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-371 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-371 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-371.1709793867.7012887.submit parsl.executors.status_handling INFO: Allocated block ID 372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-372 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-372 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-372.1709793867.701709.submit parsl.executors.status_handling INFO: Allocated block ID 373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-373 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-373 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-373.1709793867.7021484.submit parsl.executors.status_handling INFO: Allocated block ID 374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-374 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-374 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-374.1709793867.7025702.submit parsl.executors.status_handling INFO: Allocated block ID 375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-375 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-375 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-375.1709793867.7029982.submit parsl.executors.status_handling INFO: Allocated block ID 376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-376 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-376 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-376.1709793867.703411.submit parsl.executors.status_handling INFO: Allocated block ID 377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-377 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-377 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-377.1709793867.7038178.submit parsl.executors.status_handling INFO: Allocated block ID 378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-378 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-378 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-378.1709793867.7042153.submit parsl.executors.status_handling INFO: Allocated block ID 379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-379 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-379 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-379.1709793867.7046149.submit parsl.executors.status_handling INFO: Allocated block ID 380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-380 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-380 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-380.1709793867.705024.submit parsl.executors.status_handling INFO: Allocated block ID 381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-381 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-381 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-381.1709793867.7054226.submit parsl.executors.status_handling INFO: Allocated block ID 382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-382 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-382 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-382.1709793867.7058563.submit parsl.executors.status_handling INFO: Allocated block ID 383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-383 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-383 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-383.1709793867.706281.submit parsl.executors.status_handling INFO: Allocated block ID 384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-384 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-384 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-384.1709793867.7067082.submit parsl.executors.status_handling INFO: Allocated block ID 385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-385 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-385 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-385.1709793867.7071354.submit parsl.executors.status_handling INFO: Allocated block ID 386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-386 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-386 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-386.1709793867.7075384.submit parsl.executors.status_handling INFO: Allocated block ID 387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-387 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-387 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-387.1709793867.7079468.submit parsl.executors.status_handling INFO: Allocated block ID 388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-388 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-388 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-388.1709793867.708351.submit parsl.executors.status_handling INFO: Allocated block ID 389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-389 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-389 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-389.1709793867.7087574.submit parsl.executors.status_handling INFO: Allocated block ID 390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-390 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-390 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-390.1709793867.7091668.submit parsl.executors.status_handling INFO: Allocated block ID 391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-391 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-391 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-391.1709793867.7095618.submit parsl.executors.status_handling INFO: Allocated block ID 392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-392 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-392 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-392.1709793867.7099664.submit parsl.executors.status_handling INFO: Allocated block ID 393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-393 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-393 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-393.1709793867.7103665.submit parsl.executors.status_handling INFO: Allocated block ID 394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-394 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-394 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-394.1709793867.7107656.submit parsl.executors.status_handling INFO: Allocated block ID 395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-395 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-395 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-395.1709793867.711208.submit parsl.executors.status_handling INFO: Allocated block ID 396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-396 2024-03-06 22:44:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-396 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-396.1709793867.7116275.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27778179af1b346358f08b32e588cd5e with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cebe659c-b9c5-49ff-9714-b240e374f798_biasIsr_1123837_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/cebe659c-b9c5-49ff-9714-b240e374f798_biasIsr_1123837_47.stderr parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34da67861d542147a42721729950bb24 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b82063eb-abb0-4868-99d0-6c1284d28760_biasIsr_1123730_47.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123730/b82063eb-abb0-4868-99d0-6c1284d28760_biasIsr_1123730_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1362 parsl.dataflow.strategy DEBUG: Executor multi has 1362 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-397 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-397 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-397.1709793872.6261554.submit parsl.executors.status_handling INFO: Allocated block ID 398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-398 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-398 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-398.1709793872.6266434.submit parsl.executors.status_handling INFO: Allocated block ID 399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-399 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-399 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-399.1709793872.6270998.submit parsl.executors.status_handling INFO: Allocated block ID 400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-400 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-400 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-400.1709793872.627541.submit parsl.executors.status_handling INFO: Allocated block ID 401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-401 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-401 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-401.1709793872.6279786.submit parsl.executors.status_handling INFO: Allocated block ID 402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-402 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-402 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-402.1709793872.6284106.submit parsl.executors.status_handling INFO: Allocated block ID 403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-403 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-403 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-403.1709793872.628859.submit parsl.executors.status_handling INFO: Allocated block ID 404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-404 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-404 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-404.1709793872.6292863.submit parsl.executors.status_handling INFO: Allocated block ID 405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-405 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-405 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-405.1709793872.6297123.submit parsl.executors.status_handling INFO: Allocated block ID 406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-406 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-406 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-406.1709793872.6301513.submit parsl.executors.status_handling INFO: Allocated block ID 407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-407 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-407 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-407.1709793872.6305792.submit parsl.executors.status_handling INFO: Allocated block ID 408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-408 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-408 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-408.1709793872.631013.submit parsl.executors.status_handling INFO: Allocated block ID 409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-409 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-409 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-409.1709793872.631444.submit parsl.executors.status_handling INFO: Allocated block ID 410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-410 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-410 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-410.1709793872.6318734.submit parsl.executors.status_handling INFO: Allocated block ID 411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-411 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-411 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-411.1709793872.6323006.submit parsl.executors.status_handling INFO: Allocated block ID 412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-412 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-412 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-412.1709793872.6327221.submit parsl.executors.status_handling INFO: Allocated block ID 413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-413 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-413 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-413.1709793872.6331494.submit parsl.executors.status_handling INFO: Allocated block ID 414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-414 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-414 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-414.1709793872.6335683.submit parsl.executors.status_handling INFO: Allocated block ID 415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-415 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-415 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-415.1709793872.6340003.submit parsl.executors.status_handling INFO: Allocated block ID 416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-416 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-416 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-416.1709793872.6344237.submit parsl.executors.status_handling INFO: Allocated block ID 417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-417 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-417 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-417.1709793872.6348588.submit parsl.executors.status_handling INFO: Allocated block ID 418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-418 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-418 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-418.1709793872.6352837.submit parsl.executors.status_handling INFO: Allocated block ID 419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-419 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-419 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-419.1709793872.6357105.submit parsl.executors.status_handling INFO: Allocated block ID 420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-420 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-420 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-420.1709793872.6463447.submit parsl.executors.status_handling INFO: Allocated block ID 421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-421 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-421 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-421.1709793872.6467931.submit parsl.executors.status_handling INFO: Allocated block ID 422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-422 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-422 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-422.1709793872.6472263.submit parsl.executors.status_handling INFO: Allocated block ID 423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-423 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-423 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-423.1709793872.6476567.submit parsl.executors.status_handling INFO: Allocated block ID 424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-424 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-424 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-424.1709793872.648093.submit parsl.executors.status_handling INFO: Allocated block ID 425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-425 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-425 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-425.1709793872.64852.submit parsl.executors.status_handling INFO: Allocated block ID 426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-426 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-426 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-426.1709793872.648949.submit parsl.executors.status_handling INFO: Allocated block ID 427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-427 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-427 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-427.1709793872.6493783.submit parsl.executors.status_handling INFO: Allocated block ID 428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-428 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-428 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-428.1709793872.6498199.submit parsl.executors.status_handling INFO: Allocated block ID 429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-429 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-429 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-429.1709793872.650253.submit parsl.executors.status_handling INFO: Allocated block ID 430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-430 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-430 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-430.1709793872.6506822.submit parsl.executors.status_handling INFO: Allocated block ID 431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-431 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-431 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-431.1709793872.6511147.submit parsl.executors.status_handling INFO: Allocated block ID 432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-432 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-432 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-432.1709793872.6515431.submit parsl.executors.status_handling INFO: Allocated block ID 433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-433 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-433 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-433.1709793872.6519742.submit parsl.executors.status_handling INFO: Allocated block ID 434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-434 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-434 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-434.1709793872.6523983.submit parsl.executors.status_handling INFO: Allocated block ID 435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-435 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-435 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-435.1709793872.6528325.submit parsl.executors.status_handling INFO: Allocated block ID 436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-436 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-436 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-436.1709793872.6532557.submit parsl.executors.status_handling INFO: Allocated block ID 437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-437 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-437 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-437.1709793872.653676.submit parsl.executors.status_handling INFO: Allocated block ID 438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-438 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-438 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-438.1709793872.6541035.submit parsl.executors.status_handling INFO: Allocated block ID 439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-439 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-439 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-439.1709793872.6545312.submit parsl.executors.status_handling INFO: Allocated block ID 440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-440 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-440 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-440.1709793872.6549585.submit parsl.executors.status_handling INFO: Allocated block ID 441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-441 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-441 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-441.1709793872.6553805.submit parsl.executors.status_handling INFO: Allocated block ID 442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-442 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-442 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-442.1709793872.6557999.submit parsl.executors.status_handling INFO: Allocated block ID 443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-443 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-443 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-443.1709793872.6562347.submit parsl.executors.status_handling INFO: Allocated block ID 444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-444 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-444 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-444.1709793872.6566577.submit parsl.executors.status_handling INFO: Allocated block ID 445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-445 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-445 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-445.1709793872.6672661.submit parsl.executors.status_handling INFO: Allocated block ID 446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-446 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-446 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-446.1709793872.6677153.submit parsl.executors.status_handling INFO: Allocated block ID 447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-447 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-447 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-447.1709793872.6681666.submit parsl.executors.status_handling INFO: Allocated block ID 448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-448 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-448 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-448.1709793872.668597.submit parsl.executors.status_handling INFO: Allocated block ID 449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-449 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-449 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-449.1709793872.6690378.submit parsl.executors.status_handling INFO: Allocated block ID 450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-450 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-450 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-450.1709793872.669459.submit parsl.executors.status_handling INFO: Allocated block ID 451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-451 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-451 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-451.1709793872.6698942.submit parsl.executors.status_handling INFO: Allocated block ID 452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-452 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-452 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-452.1709793872.6703196.submit parsl.executors.status_handling INFO: Allocated block ID 453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-453 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-453 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-453.1709793872.6707404.submit parsl.executors.status_handling INFO: Allocated block ID 454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-454 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-454 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-454.1709793872.6711693.submit parsl.executors.status_handling INFO: Allocated block ID 455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-455 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-455 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-455.1709793872.6715918.submit parsl.executors.status_handling INFO: Allocated block ID 456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-456 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-456 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-456.1709793872.672017.submit parsl.executors.status_handling INFO: Allocated block ID 457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-457 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-457 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-457.1709793872.6724405.submit parsl.executors.status_handling INFO: Allocated block ID 458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-458 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-458 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-458.1709793872.67287.submit parsl.executors.status_handling INFO: Allocated block ID 459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-459 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-459 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-459.1709793872.6733065.submit parsl.executors.status_handling INFO: Allocated block ID 460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-460 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-460 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-460.1709793872.6737301.submit parsl.executors.status_handling INFO: Allocated block ID 461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-461 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-461 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-461.1709793872.6741633.submit parsl.executors.status_handling INFO: Allocated block ID 462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-462 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-462 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-462.1709793872.6745844.submit parsl.executors.status_handling INFO: Allocated block ID 463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-463 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-463 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-463.1709793872.6750174.submit parsl.executors.status_handling INFO: Allocated block ID 464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-464 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-464 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-464.1709793872.6754413.submit parsl.executors.status_handling INFO: Allocated block ID 465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-465 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-465 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-465.1709793872.6758666.submit parsl.executors.status_handling INFO: Allocated block ID 466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-466 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-466 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-466.1709793872.6762905.submit parsl.executors.status_handling INFO: Allocated block ID 467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-467 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-467 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-467.1709793872.6767104.submit parsl.executors.status_handling INFO: Allocated block ID 468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-468 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-468 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-468.1709793872.6771455.submit parsl.executors.status_handling INFO: Allocated block ID 469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-469 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-469 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-469.1709793872.6775742.submit parsl.executors.status_handling INFO: Allocated block ID 470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-470 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-470 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-470.1709793872.6881812.submit parsl.executors.status_handling INFO: Allocated block ID 471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-471 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-471 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-471.1709793872.6886477.submit parsl.executors.status_handling INFO: Allocated block ID 472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-472 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-472 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-472.1709793872.689089.submit parsl.executors.status_handling INFO: Allocated block ID 473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-473 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-473 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-473.1709793872.6895201.submit parsl.executors.status_handling INFO: Allocated block ID 474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-474 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-474 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-474.1709793872.6899543.submit parsl.executors.status_handling INFO: Allocated block ID 475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-475 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-475 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-475.1709793872.690389.submit parsl.executors.status_handling INFO: Allocated block ID 476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-476 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-476 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-476.1709793872.6908228.submit parsl.executors.status_handling INFO: Allocated block ID 477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-477 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-477 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-477.1709793872.6912618.submit parsl.executors.status_handling INFO: Allocated block ID 478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-478 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-478 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-478.1709793872.6916852.submit parsl.executors.status_handling INFO: Allocated block ID 479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-479 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-479 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-479.1709793872.6921203.submit parsl.executors.status_handling INFO: Allocated block ID 480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-480 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-480 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-480.1709793872.6925504.submit parsl.executors.status_handling INFO: Allocated block ID 481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-481 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-481 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-481.1709793872.6929884.submit parsl.executors.status_handling INFO: Allocated block ID 482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-482 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-482 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-482.1709793872.693415.submit parsl.executors.status_handling INFO: Allocated block ID 483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-483 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-483 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-483.1709793872.6938584.submit parsl.executors.status_handling INFO: Allocated block ID 484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-484 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-484 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-484.1709793872.69429.submit parsl.executors.status_handling INFO: Allocated block ID 485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-485 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-485 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-485.1709793872.6947167.submit parsl.executors.status_handling INFO: Allocated block ID 486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-486 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-486 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-486.1709793872.6951551.submit parsl.executors.status_handling INFO: Allocated block ID 487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-487 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-487 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-487.1709793872.695588.submit parsl.executors.status_handling INFO: Allocated block ID 488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-488 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-488 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-488.1709793872.6960225.submit parsl.executors.status_handling INFO: Allocated block ID 489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-489 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-489 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-489.1709793872.6964493.submit parsl.executors.status_handling INFO: Allocated block ID 490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-490 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-490 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-490.1709793872.7082167.submit parsl.executors.status_handling INFO: Allocated block ID 491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-491 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-491 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-491.1709793872.7087033.submit parsl.executors.status_handling INFO: Allocated block ID 492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-492 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-492 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-492.1709793872.7091491.submit parsl.executors.status_handling INFO: Allocated block ID 493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-493 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-493 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-493.1709793872.7095795.submit parsl.executors.status_handling INFO: Allocated block ID 494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-494 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-494 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-494.1709793872.7100134.submit parsl.executors.status_handling INFO: Allocated block ID 495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-495 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-495 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-495.1709793872.710444.submit parsl.executors.status_handling INFO: Allocated block ID 496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-496 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-496 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-496.1709793872.7108762.submit parsl.executors.status_handling INFO: Allocated block ID 497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-497 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-497 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-497.1709793872.7113032.submit parsl.executors.status_handling INFO: Allocated block ID 498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-498 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-498 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-498.1709793872.7117274.submit parsl.executors.status_handling INFO: Allocated block ID 499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-499 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-499 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-499.1709793872.7121606.submit parsl.executors.status_handling INFO: Allocated block ID 500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-500 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-500 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-500.1709793872.712586.submit parsl.executors.status_handling INFO: Allocated block ID 501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-501 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-501 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-501.1709793872.713015.submit parsl.executors.status_handling INFO: Allocated block ID 502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-502 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-502 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-502.1709793872.713438.submit parsl.executors.status_handling INFO: Allocated block ID 503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-503 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-503 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-503.1709793872.713869.submit parsl.executors.status_handling INFO: Allocated block ID 504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-504 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-504 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-504.1709793872.714296.submit parsl.executors.status_handling INFO: Allocated block ID 505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-505 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-505 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-505.1709793872.7147245.submit parsl.executors.status_handling INFO: Allocated block ID 506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-506 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-506 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-506.1709793872.7151606.submit parsl.executors.status_handling INFO: Allocated block ID 507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-507 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-507 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-507.1709793872.7155962.submit parsl.executors.status_handling INFO: Allocated block ID 508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-508 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-508 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-508.1709793872.7160254.submit parsl.executors.status_handling INFO: Allocated block ID 509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-509 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-509 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-509.1709793872.7164533.submit parsl.executors.status_handling INFO: Allocated block ID 510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-510 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-510 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-510.1709793872.7168849.submit parsl.executors.status_handling INFO: Allocated block ID 511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-511 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-511 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-511.1709793872.7173128.submit parsl.executors.status_handling INFO: Allocated block ID 512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-512 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-512 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-512.1709793872.717734.submit parsl.executors.status_handling INFO: Allocated block ID 513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-513 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-513 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-513.1709793872.718185.submit parsl.executors.status_handling INFO: Allocated block ID 514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-514 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-514 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-514.1709793872.7186167.submit parsl.executors.status_handling INFO: Allocated block ID 515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-515 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-515 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-515.1709793872.7292492.submit parsl.executors.status_handling INFO: Allocated block ID 516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-516 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-516 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-516.1709793872.729713.submit parsl.executors.status_handling INFO: Allocated block ID 517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-517 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-517 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-517.1709793872.7301643.submit parsl.executors.status_handling INFO: Allocated block ID 518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-518 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-518 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-518.1709793872.7306015.submit parsl.executors.status_handling INFO: Allocated block ID 519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-519 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-519 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-519.1709793872.73105.submit parsl.executors.status_handling INFO: Allocated block ID 520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-520 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-520 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-520.1709793872.7314906.submit parsl.executors.status_handling INFO: Allocated block ID 521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-521 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-521 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-521.1709793872.7319343.submit parsl.executors.status_handling INFO: Allocated block ID 522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-522 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-522 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-522.1709793872.7323844.submit parsl.executors.status_handling INFO: Allocated block ID 523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-523 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-523 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-523.1709793872.732832.submit parsl.executors.status_handling INFO: Allocated block ID 524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-524 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-524 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-524.1709793872.7332668.submit parsl.executors.status_handling INFO: Allocated block ID 525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-525 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-525 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-525.1709793872.7337008.submit parsl.executors.status_handling INFO: Allocated block ID 526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-526 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-526 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-526.1709793872.7341316.submit parsl.executors.status_handling INFO: Allocated block ID 527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-527 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-527 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-527.1709793872.7345545.submit parsl.executors.status_handling INFO: Allocated block ID 528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-528 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-528 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-528.1709793872.734983.submit parsl.executors.status_handling INFO: Allocated block ID 529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-529 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-529 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-529.1709793872.7354085.submit parsl.executors.status_handling INFO: Allocated block ID 530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-530 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-530 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-530.1709793872.7358367.submit parsl.executors.status_handling INFO: Allocated block ID 531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-531 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-531 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-531.1709793872.7362654.submit parsl.executors.status_handling INFO: Allocated block ID 532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-532 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-532 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-532.1709793872.736686.submit parsl.executors.status_handling INFO: Allocated block ID 533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-533 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-533 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-533.1709793872.7371104.submit parsl.executors.status_handling INFO: Allocated block ID 534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-534 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-534 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-534.1709793872.7375433.submit parsl.executors.status_handling INFO: Allocated block ID 535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-535 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-535 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-535.1709793872.7379854.submit parsl.executors.status_handling INFO: Allocated block ID 536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-536 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-536 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-536.1709793872.738416.submit parsl.executors.status_handling INFO: Allocated block ID 537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-537 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-537 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-537.1709793872.7388523.submit parsl.executors.status_handling INFO: Allocated block ID 538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-538 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-538 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-538.1709793872.7392874.submit parsl.executors.status_handling INFO: Allocated block ID 539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-539 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-539 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-539.1709793872.739713.submit parsl.executors.status_handling INFO: Allocated block ID 540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-540 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-540 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-540.1709793872.740181.submit parsl.executors.status_handling INFO: Allocated block ID 541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-541 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-541 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-541.1709793872.7406104.submit parsl.executors.status_handling INFO: Allocated block ID 542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-542 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-542 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-542.1709793872.7410748.submit parsl.executors.status_handling INFO: Allocated block ID 543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-543 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-543 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-543.1709793872.741515.submit parsl.executors.status_handling INFO: Allocated block ID 544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-544 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-544 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-544.1709793872.74196.submit parsl.executors.status_handling INFO: Allocated block ID 545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-545 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-545 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-545.1709793872.7423804.submit parsl.executors.status_handling INFO: Allocated block ID 546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-546 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-546 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-546.1709793872.742803.submit parsl.executors.status_handling INFO: Allocated block ID 547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-547 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-547 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-547.1709793872.7432377.submit parsl.executors.status_handling INFO: Allocated block ID 548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-548 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-548 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-548.1709793872.7436552.submit parsl.executors.status_handling INFO: Allocated block ID 549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-549 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-549 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-549.1709793872.7440815.submit parsl.executors.status_handling INFO: Allocated block ID 550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-550 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-550 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-550.1709793872.744504.submit parsl.executors.status_handling INFO: Allocated block ID 551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-551 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-551 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-551.1709793872.7449343.submit parsl.executors.status_handling INFO: Allocated block ID 552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-552 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-552 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-552.1709793872.7453647.submit parsl.executors.status_handling INFO: Allocated block ID 553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-553 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-553 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-553.1709793872.745795.submit parsl.executors.status_handling INFO: Allocated block ID 554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-554 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-554 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-554.1709793872.7462397.submit parsl.executors.status_handling INFO: Allocated block ID 555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-555 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-555 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-555.1709793872.7466884.submit parsl.executors.status_handling INFO: Allocated block ID 556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-556 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-556 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-556.1709793872.7471414.submit parsl.executors.status_handling INFO: Allocated block ID 557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-557 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-557 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-557.1709793872.747586.submit parsl.executors.status_handling INFO: Allocated block ID 558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-558 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-558 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-558.1709793872.7480137.submit parsl.executors.status_handling INFO: Allocated block ID 559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-559 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-559 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-559.1709793872.7484436.submit parsl.executors.status_handling INFO: Allocated block ID 560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-560 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-560 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-560.1709793872.748885.submit parsl.executors.status_handling INFO: Allocated block ID 561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-561 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-561 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-561.1709793872.7493117.submit parsl.executors.status_handling INFO: Allocated block ID 562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-562 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-562 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-562.1709793872.7497344.submit parsl.executors.status_handling INFO: Allocated block ID 563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-563 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-563 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-563.1709793872.7501683.submit parsl.executors.status_handling INFO: Allocated block ID 564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-564 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-564 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-564.1709793872.7505908.submit parsl.executors.status_handling INFO: Allocated block ID 565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-565 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-565 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-565.1709793872.751058.submit parsl.executors.status_handling INFO: Allocated block ID 566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-566 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-566 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-566.1709793872.751493.submit parsl.executors.status_handling INFO: Allocated block ID 567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-567 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-567 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-567.1709793872.7519457.submit parsl.executors.status_handling INFO: Allocated block ID 568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-568 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-568 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-568.1709793872.7523699.submit parsl.executors.status_handling INFO: Allocated block ID 569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-569 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-569 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-569.1709793872.7527962.submit parsl.executors.status_handling INFO: Allocated block ID 570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-570 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-570 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-570.1709793872.7532291.submit parsl.executors.status_handling INFO: Allocated block ID 571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-571 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-571 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-571.1709793872.7536635.submit parsl.executors.status_handling INFO: Allocated block ID 572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-572 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-572 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-572.1709793872.7540956.submit parsl.executors.status_handling INFO: Allocated block ID 573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-573 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-573 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-573.1709793872.754525.submit parsl.executors.status_handling INFO: Allocated block ID 574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-574 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-574 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-574.1709793872.7549634.submit parsl.executors.status_handling INFO: Allocated block ID 575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-575 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-575 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-575.1709793872.7553859.submit parsl.executors.status_handling INFO: Allocated block ID 576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-576 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-576 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-576.1709793872.7558231.submit parsl.executors.status_handling INFO: Allocated block ID 577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-577 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-577 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-577.1709793872.756258.submit parsl.executors.status_handling INFO: Allocated block ID 578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-578 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-578 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-578.1709793872.7566857.submit parsl.executors.status_handling INFO: Allocated block ID 579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-579 2024-03-06 22:44:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-579 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-579.1709793872.757125.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1362 parsl.dataflow.strategy DEBUG: Executor multi has 1362 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-580 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-580 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-580.1709793877.6198187.submit parsl.executors.status_handling INFO: Allocated block ID 581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-581 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-581 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-581.1709793877.6219375.submit parsl.executors.status_handling INFO: Allocated block ID 582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-582 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-582 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-582.1709793877.6237886.submit parsl.executors.status_handling INFO: Allocated block ID 583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-583 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-583 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-583.1709793877.6252613.submit parsl.executors.status_handling INFO: Allocated block ID 584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-584 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-584 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-584.1709793877.6267018.submit parsl.executors.status_handling INFO: Allocated block ID 585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-585 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-585 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-585.1709793877.6281538.submit parsl.executors.status_handling INFO: Allocated block ID 586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-586 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-586 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-586.1709793877.6295757.submit parsl.executors.status_handling INFO: Allocated block ID 587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-587 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-587 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-587.1709793877.6310234.submit parsl.executors.status_handling INFO: Allocated block ID 588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-588 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-588 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-588.1709793877.6324396.submit parsl.executors.status_handling INFO: Allocated block ID 589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-589 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-589 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-589.1709793877.633547.submit parsl.executors.status_handling INFO: Allocated block ID 590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-590 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-590 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-590.1709793877.6341276.submit parsl.executors.status_handling INFO: Allocated block ID 591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-591 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-591 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-591.1709793877.6347032.submit parsl.executors.status_handling INFO: Allocated block ID 592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-592 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-592 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-592.1709793877.6352863.submit parsl.executors.status_handling INFO: Allocated block ID 593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-593 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-593 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-593.1709793877.6358738.submit parsl.executors.status_handling INFO: Allocated block ID 594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-594 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-594 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-594.1709793877.6364574.submit parsl.executors.status_handling INFO: Allocated block ID 595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-595 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-595 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-595.1709793877.6370416.submit parsl.executors.status_handling INFO: Allocated block ID 596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-596 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-596 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-596.1709793877.6376204.submit parsl.executors.status_handling INFO: Allocated block ID 597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-597 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-597 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-597.1709793877.6382086.submit parsl.executors.status_handling INFO: Allocated block ID 598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-598 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-598 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-598.1709793877.6387963.submit parsl.executors.status_handling INFO: Allocated block ID 599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-599 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-599 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-599.1709793877.639393.submit parsl.executors.status_handling INFO: Allocated block ID 600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-600 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-600 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-600.1709793877.6399791.submit parsl.executors.status_handling INFO: Allocated block ID 601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-601 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-601 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-601.1709793877.6405594.submit parsl.executors.status_handling INFO: Allocated block ID 602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-602 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-602 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-602.1709793877.6411502.submit parsl.executors.status_handling INFO: Allocated block ID 603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-603 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-603 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-603.1709793877.641735.submit parsl.executors.status_handling INFO: Allocated block ID 604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-604 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-604 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-604.1709793877.6423266.submit parsl.executors.status_handling INFO: Allocated block ID 605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-605 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-605 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-605.1709793877.6429176.submit parsl.executors.status_handling INFO: Allocated block ID 606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-606 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-606 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-606.1709793877.6435046.submit parsl.executors.status_handling INFO: Allocated block ID 607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-607 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-607 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-607.1709793877.643983.submit parsl.executors.status_handling INFO: Allocated block ID 608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-608 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-608 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-608.1709793877.6444178.submit parsl.executors.status_handling INFO: Allocated block ID 609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-609 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-609 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-609.1709793877.6448636.submit parsl.executors.status_handling INFO: Allocated block ID 610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-610 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-610 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-610.1709793877.6452854.submit parsl.executors.status_handling INFO: Allocated block ID 611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-611 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-611 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-611.1709793877.6457133.submit parsl.executors.status_handling INFO: Allocated block ID 612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-612 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-612 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-612.1709793877.6461473.submit parsl.executors.status_handling INFO: Allocated block ID 613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-613 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-613 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-613.1709793877.6465695.submit parsl.executors.status_handling INFO: Allocated block ID 614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-614 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-614 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-614.1709793877.6470015.submit parsl.executors.status_handling INFO: Allocated block ID 615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-615 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-615 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-615.1709793877.6474237.submit parsl.executors.status_handling INFO: Allocated block ID 616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-616 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-616 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-616.1709793877.6478517.submit parsl.executors.status_handling INFO: Allocated block ID 617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-617 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-617 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-617.1709793877.648277.submit parsl.executors.status_handling INFO: Allocated block ID 618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-618 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-618 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-618.1709793877.6487062.submit parsl.executors.status_handling INFO: Allocated block ID 619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-619 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-619 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-619.1709793877.649151.submit parsl.executors.status_handling INFO: Allocated block ID 620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-620 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-620 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-620.1709793877.6495757.submit parsl.executors.status_handling INFO: Allocated block ID 621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-621 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-621 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-621.1709793877.6500032.submit parsl.executors.status_handling INFO: Allocated block ID 622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-622 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-622 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-622.1709793877.6504252.submit parsl.executors.status_handling INFO: Allocated block ID 623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-623 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-623 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-623.1709793877.6508539.submit parsl.executors.status_handling INFO: Allocated block ID 624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-624 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-624 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-624.1709793877.6512783.submit parsl.executors.status_handling INFO: Allocated block ID 625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-625 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-625 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-625.1709793877.6516984.submit parsl.executors.status_handling INFO: Allocated block ID 626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-626 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-626 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-626.1709793877.6521556.submit parsl.executors.status_handling INFO: Allocated block ID 627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-627 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-627 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-627.1709793877.6525955.submit parsl.executors.status_handling INFO: Allocated block ID 628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-628 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-628 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-628.1709793877.6530402.submit parsl.executors.status_handling INFO: Allocated block ID 629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-629 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-629 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-629.1709793877.6534815.submit parsl.executors.status_handling INFO: Allocated block ID 630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-630 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-630 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-630.1709793877.653916.submit parsl.executors.status_handling INFO: Allocated block ID 631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-631 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-631 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-631.1709793877.654344.submit parsl.executors.status_handling INFO: Allocated block ID 632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-632 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-632 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-632.1709793877.6547647.submit parsl.executors.status_handling INFO: Allocated block ID 633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-633 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-633 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-633.1709793877.655194.submit parsl.executors.status_handling INFO: Allocated block ID 634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-634 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-634 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-634.1709793877.6556132.submit parsl.executors.status_handling INFO: Allocated block ID 635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-635 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-635 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-635.1709793877.65605.submit parsl.executors.status_handling INFO: Allocated block ID 636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-636 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-636 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-636.1709793877.6564753.submit parsl.executors.status_handling INFO: Allocated block ID 637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-637 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-637 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-637.1709793877.6570241.submit parsl.executors.status_handling INFO: Allocated block ID 638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-638 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-638 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-638.1709793877.6574678.submit parsl.executors.status_handling INFO: Allocated block ID 639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-639 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-639 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-639.1709793877.657908.submit parsl.executors.status_handling INFO: Allocated block ID 640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-640 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-640 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-640.1709793877.6583567.submit parsl.executors.status_handling INFO: Allocated block ID 641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-641 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-641 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-641.1709793877.6587837.submit parsl.executors.status_handling INFO: Allocated block ID 642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-642 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-642 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-642.1709793877.6592195.submit parsl.executors.status_handling INFO: Allocated block ID 643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-643 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-643 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-643.1709793877.6596496.submit parsl.executors.status_handling INFO: Allocated block ID 644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-644 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-644 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-644.1709793877.6600814.submit parsl.executors.status_handling INFO: Allocated block ID 645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-645 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-645 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-645.1709793877.660507.submit parsl.executors.status_handling INFO: Allocated block ID 646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-646 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-646 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-646.1709793877.6609364.submit parsl.executors.status_handling INFO: Allocated block ID 647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-647 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-647 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-647.1709793877.6613617.submit parsl.executors.status_handling INFO: Allocated block ID 648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-648 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-648 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-648.1709793877.661783.submit parsl.executors.status_handling INFO: Allocated block ID 649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-649 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-649 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-649.1709793877.6622117.submit parsl.executors.status_handling INFO: Allocated block ID 650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-650 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-650 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-650.1709793877.6626315.submit parsl.executors.status_handling INFO: Allocated block ID 651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-651 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-651 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-651.1709793877.6630564.submit parsl.executors.status_handling INFO: Allocated block ID 652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-652 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-652 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-652.1709793877.6634786.submit parsl.executors.status_handling INFO: Allocated block ID 653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-653 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-653 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-653.1709793877.6639118.submit parsl.executors.status_handling INFO: Allocated block ID 654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-654 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-654 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-654.1709793877.6643455.submit parsl.executors.status_handling INFO: Allocated block ID 655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-655 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-655 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-655.1709793877.6647701.submit parsl.executors.status_handling INFO: Allocated block ID 656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-656 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-656 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-656.1709793877.6652007.submit parsl.executors.status_handling INFO: Allocated block ID 657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-657 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-657 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-657.1709793877.6656268.submit parsl.executors.status_handling INFO: Allocated block ID 658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-658 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-658 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-658.1709793877.666052.submit parsl.executors.status_handling INFO: Allocated block ID 659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-659 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-659 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-659.1709793877.6664786.submit parsl.executors.status_handling INFO: Allocated block ID 660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-660 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-660 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-660.1709793877.6669078.submit parsl.executors.status_handling INFO: Allocated block ID 661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-661 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-661 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-661.1709793877.6673338.submit parsl.executors.status_handling INFO: Allocated block ID 662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-662 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-662 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-662.1709793877.6677604.submit parsl.executors.status_handling INFO: Allocated block ID 663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-663 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-663 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-663.1709793877.668191.submit parsl.executors.status_handling INFO: Allocated block ID 664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-664 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-664 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-664.1709793877.668639.submit parsl.executors.status_handling INFO: Allocated block ID 665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-665 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-665 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-665.1709793877.6691139.submit parsl.executors.status_handling INFO: Allocated block ID 666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-666 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-666 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-666.1709793877.6695774.submit parsl.executors.status_handling INFO: Allocated block ID 667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-667 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-667 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-667.1709793877.6700194.submit parsl.executors.status_handling INFO: Allocated block ID 668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-668 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-668 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-668.1709793877.6704566.submit parsl.executors.status_handling INFO: Allocated block ID 669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-669 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-669 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-669.1709793877.6708837.submit parsl.executors.status_handling INFO: Allocated block ID 670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-670 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-670 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-670.1709793877.671308.submit parsl.executors.status_handling INFO: Allocated block ID 671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-671 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-671 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-671.1709793877.6717322.submit parsl.executors.status_handling INFO: Allocated block ID 672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-672 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-672 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-672.1709793877.6721687.submit parsl.executors.status_handling INFO: Allocated block ID 673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-673 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-673 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-673.1709793877.6725938.submit parsl.executors.status_handling INFO: Allocated block ID 674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-674 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-674 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-674.1709793877.6730206.submit parsl.executors.status_handling INFO: Allocated block ID 675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-675 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-675 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-675.1709793877.6734428.submit parsl.executors.status_handling INFO: Allocated block ID 676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-676 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-676 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-676.1709793877.6738732.submit parsl.executors.status_handling INFO: Allocated block ID 677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-677 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-677 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-677.1709793877.6743014.submit parsl.executors.status_handling INFO: Allocated block ID 678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-678 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-678 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-678.1709793877.6747272.submit parsl.executors.status_handling INFO: Allocated block ID 679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-679 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-679 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-679.1709793877.6751578.submit parsl.executors.status_handling INFO: Allocated block ID 680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-680 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-680 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-680.1709793877.6755798.submit parsl.executors.status_handling INFO: Allocated block ID 681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-681 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-681 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-681.1709793877.6760101.submit parsl.executors.status_handling INFO: Allocated block ID 682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-682 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-682 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-682.1709793877.676432.submit parsl.executors.status_handling INFO: Allocated block ID 683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-683 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-683 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-683.1709793877.6768706.submit parsl.executors.status_handling INFO: Allocated block ID 684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-684 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-684 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-684.1709793877.6773417.submit parsl.executors.status_handling INFO: Allocated block ID 685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-685 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-685 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-685.1709793877.6777694.submit parsl.executors.status_handling INFO: Allocated block ID 686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-686 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-686 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-686.1709793877.6781988.submit parsl.executors.status_handling INFO: Allocated block ID 687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-687 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-687 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-687.1709793877.6786308.submit parsl.executors.status_handling INFO: Allocated block ID 688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-688 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-688 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-688.1709793877.6790607.submit parsl.executors.status_handling INFO: Allocated block ID 689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-689 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-689 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-689.1709793877.6794882.submit parsl.executors.status_handling INFO: Allocated block ID 690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-690 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-690 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-690.1709793877.6799247.submit parsl.executors.status_handling INFO: Allocated block ID 691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-691 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-691 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-691.1709793877.6803687.submit parsl.executors.status_handling INFO: Allocated block ID 692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-692 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-692 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-692.1709793877.6808138.submit parsl.executors.status_handling INFO: Allocated block ID 693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-693 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-693 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-693.1709793877.6812499.submit parsl.executors.status_handling INFO: Allocated block ID 694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-694 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-694 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-694.1709793877.6816833.submit parsl.executors.status_handling INFO: Allocated block ID 695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-695 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-695 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-695.1709793877.6821268.submit parsl.executors.status_handling INFO: Allocated block ID 696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-696 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-696 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-696.1709793877.6825604.submit parsl.executors.status_handling INFO: Allocated block ID 697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-697 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-697 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-697.1709793877.6830099.submit parsl.executors.status_handling INFO: Allocated block ID 698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-698 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-698 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-698.1709793877.6834316.submit parsl.executors.status_handling INFO: Allocated block ID 699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-699 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-699 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-699.1709793877.6838608.submit parsl.executors.status_handling INFO: Allocated block ID 700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-700 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-700 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-700.1709793877.6842825.submit parsl.executors.status_handling INFO: Allocated block ID 701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-701 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-701 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-701.1709793877.684708.submit parsl.executors.status_handling INFO: Allocated block ID 702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-702 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-702 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-702.1709793877.6851437.submit parsl.executors.status_handling INFO: Allocated block ID 703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-703 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-703 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-703.1709793877.6855736.submit parsl.executors.status_handling INFO: Allocated block ID 704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-704 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-704 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-704.1709793877.6860092.submit parsl.executors.status_handling INFO: Allocated block ID 705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-705 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-705 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-705.1709793877.6864326.submit parsl.executors.status_handling INFO: Allocated block ID 706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-706 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-706 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-706.1709793877.6868584.submit parsl.executors.status_handling INFO: Allocated block ID 707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-707 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-707 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-707.1709793877.687287.submit parsl.executors.status_handling INFO: Allocated block ID 708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-708 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-708 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-708.1709793877.687708.submit parsl.executors.status_handling INFO: Allocated block ID 709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-709 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-709 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-709.1709793877.6881409.submit parsl.executors.status_handling INFO: Allocated block ID 710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-710 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-710 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-710.1709793877.6885617.submit parsl.executors.status_handling INFO: Allocated block ID 711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-711 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-711 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-711.1709793877.688989.submit parsl.executors.status_handling INFO: Allocated block ID 712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-712 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-712 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-712.1709793877.6894138.submit parsl.executors.status_handling INFO: Allocated block ID 713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-713 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-713 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-713.1709793877.6898456.submit parsl.executors.status_handling INFO: Allocated block ID 714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-714 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-714 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-714.1709793877.6902719.submit parsl.executors.status_handling INFO: Allocated block ID 715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-715 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-715 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-715.1709793877.6906946.submit parsl.executors.status_handling INFO: Allocated block ID 716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-716 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-716 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-716.1709793877.691124.submit parsl.executors.status_handling INFO: Allocated block ID 717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-717 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-717 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-717.1709793877.6915505.submit parsl.executors.status_handling INFO: Allocated block ID 718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-718 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-718 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-718.1709793877.6919801.submit parsl.executors.status_handling INFO: Allocated block ID 719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-719 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-719 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-719.1709793877.6924078.submit parsl.executors.status_handling INFO: Allocated block ID 720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-720 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-720 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-720.1709793877.692835.submit parsl.executors.status_handling INFO: Allocated block ID 721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-721 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-721 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-721.1709793877.6932545.submit parsl.executors.status_handling INFO: Allocated block ID 722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-722 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-722 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-722.1709793877.693675.submit parsl.executors.status_handling INFO: Allocated block ID 723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-723 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-723 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-723.1709793877.694101.submit parsl.executors.status_handling INFO: Allocated block ID 724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-724 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-724 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-724.1709793877.6945195.submit parsl.executors.status_handling INFO: Allocated block ID 725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-725 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-725 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-725.1709793877.6949594.submit parsl.executors.status_handling INFO: Allocated block ID 726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-726 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-726 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-726.1709793877.6953886.submit parsl.executors.status_handling INFO: Allocated block ID 727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-727 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-727 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-727.1709793877.69582.submit parsl.executors.status_handling INFO: Allocated block ID 728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-728 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-728 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-728.1709793877.6962483.submit parsl.executors.status_handling INFO: Allocated block ID 729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-729 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-729 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-729.1709793877.696672.submit parsl.executors.status_handling INFO: Allocated block ID 730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-730 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-730 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-730.1709793877.6971014.submit parsl.executors.status_handling INFO: Allocated block ID 731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-731 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-731 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-731.1709793877.6975284.submit parsl.executors.status_handling INFO: Allocated block ID 732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-732 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-732 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-732.1709793877.6979606.submit parsl.executors.status_handling INFO: Allocated block ID 733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-733 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-733 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-733.1709793877.698386.submit parsl.executors.status_handling INFO: Allocated block ID 734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-734 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-734 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-734.1709793877.6988027.submit parsl.executors.status_handling INFO: Allocated block ID 735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-735 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-735 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-735.1709793877.6992328.submit parsl.executors.status_handling INFO: Allocated block ID 736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-736 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-736 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-736.1709793877.6996524.submit parsl.executors.status_handling INFO: Allocated block ID 737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-737 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-737 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-737.1709793877.7000859.submit parsl.executors.status_handling INFO: Allocated block ID 738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-738 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-738 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-738.1709793877.7005155.submit parsl.executors.status_handling INFO: Allocated block ID 739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-739 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-739 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-739.1709793877.7010574.submit parsl.executors.status_handling INFO: Allocated block ID 740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-740 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-740 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-740.1709793877.7014914.submit parsl.executors.status_handling INFO: Allocated block ID 741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-741 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-741 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-741.1709793877.7019224.submit parsl.executors.status_handling INFO: Allocated block ID 742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-742 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-742 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-742.1709793877.7023494.submit parsl.executors.status_handling INFO: Allocated block ID 743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-743 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-743 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-743.1709793877.7027743.submit parsl.executors.status_handling INFO: Allocated block ID 744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-744 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-744 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-744.1709793877.7032037.submit parsl.executors.status_handling INFO: Allocated block ID 745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-745 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-745 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-745.1709793877.703624.submit parsl.executors.status_handling INFO: Allocated block ID 746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-746 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-746 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-746.1709793877.704056.submit parsl.executors.status_handling INFO: Allocated block ID 747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-747 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-747 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-747.1709793877.7044995.submit parsl.executors.status_handling INFO: Allocated block ID 748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-748 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-748 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-748.1709793877.7049537.submit parsl.executors.status_handling INFO: Allocated block ID 749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-749 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-749 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-749.1709793877.7053986.submit parsl.executors.status_handling INFO: Allocated block ID 750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-750 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-750 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-750.1709793877.7058551.submit parsl.executors.status_handling INFO: Allocated block ID 751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-751 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-751 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-751.1709793877.7062917.submit parsl.executors.status_handling INFO: Allocated block ID 752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-752 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-752 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-752.1709793877.7067277.submit parsl.executors.status_handling INFO: Allocated block ID 753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-753 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-753 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-753.1709793877.7071595.submit parsl.executors.status_handling INFO: Allocated block ID 754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-754 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-754 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-754.1709793877.707587.submit parsl.executors.status_handling INFO: Allocated block ID 755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-755 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-755 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-755.1709793877.7080274.submit parsl.executors.status_handling INFO: Allocated block ID 756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-756 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-756 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-756.1709793877.7084656.submit parsl.executors.status_handling INFO: Allocated block ID 757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-757 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-757 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-757.1709793877.7089262.submit parsl.executors.status_handling INFO: Allocated block ID 758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-758 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-758 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-758.1709793877.709472.submit parsl.executors.status_handling INFO: Allocated block ID 759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-759 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-759 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-759.1709793877.7099617.submit parsl.executors.status_handling INFO: Allocated block ID 760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-760 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-760 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-760.1709793877.7104154.submit parsl.executors.status_handling INFO: Allocated block ID 761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-761 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-761 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-761.1709793877.7108727.submit parsl.executors.status_handling INFO: Allocated block ID 762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-762 2024-03-06 22:44:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-762 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-762.1709793877.7113552.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1362 parsl.dataflow.strategy DEBUG: Executor multi has 1362 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-763 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-763 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-763.1709793882.6198115.submit parsl.executors.status_handling INFO: Allocated block ID 764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-764 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-764 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-764.1709793882.6203141.submit parsl.executors.status_handling INFO: Allocated block ID 765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-765 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-765 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-765.1709793882.6207566.submit parsl.executors.status_handling INFO: Allocated block ID 766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-766 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-766 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-766.1709793882.6212037.submit parsl.executors.status_handling INFO: Allocated block ID 767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-767 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-767 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-767.1709793882.621635.submit parsl.executors.status_handling INFO: Allocated block ID 768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-768 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-768 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-768.1709793882.6220777.submit parsl.executors.status_handling INFO: Allocated block ID 769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-769 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-769 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-769.1709793882.6225092.submit parsl.executors.status_handling INFO: Allocated block ID 770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-770 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-770 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-770.1709793882.6229415.submit parsl.executors.status_handling INFO: Allocated block ID 771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-771 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-771 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-771.1709793882.6233733.submit parsl.executors.status_handling INFO: Allocated block ID 772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-772 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-772 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-772.1709793882.6237996.submit parsl.executors.status_handling INFO: Allocated block ID 773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-773 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-773 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-773.1709793882.6242335.submit parsl.executors.status_handling INFO: Allocated block ID 774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-774 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-774 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-774.1709793882.6246612.submit parsl.executors.status_handling INFO: Allocated block ID 775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-775 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-775 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-775.1709793882.6250916.submit parsl.executors.status_handling INFO: Allocated block ID 776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-776 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-776 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-776.1709793882.6255143.submit parsl.executors.status_handling INFO: Allocated block ID 777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-777 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-777 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-777.1709793882.6259604.submit parsl.executors.status_handling INFO: Allocated block ID 778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-778 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-778 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-778.1709793882.6263857.submit parsl.executors.status_handling INFO: Allocated block ID 779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-779 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-779 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-779.1709793882.6268134.submit parsl.executors.status_handling INFO: Allocated block ID 780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-780 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-780 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-780.1709793882.6272361.submit parsl.executors.status_handling INFO: Allocated block ID 781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-781 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-781 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-781.1709793882.6276588.submit parsl.executors.status_handling INFO: Allocated block ID 782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-782 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-782 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-782.1709793882.6280878.submit parsl.executors.status_handling INFO: Allocated block ID 783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-783 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-783 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-783.1709793882.6285155.submit parsl.executors.status_handling INFO: Allocated block ID 784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-784 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-784 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-784.1709793882.6289485.submit parsl.executors.status_handling INFO: Allocated block ID 785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-785 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-785 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-785.1709793882.6293705.submit parsl.executors.status_handling INFO: Allocated block ID 786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-786 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-786 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-786.1709793882.629792.submit parsl.executors.status_handling INFO: Allocated block ID 787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-787 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-787 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-787.1709793882.6302848.submit parsl.executors.status_handling INFO: Allocated block ID 788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-788 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-788 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-788.1709793882.6307077.submit parsl.executors.status_handling INFO: Allocated block ID 789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-789 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-789 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-789.1709793882.6311553.submit parsl.executors.status_handling INFO: Allocated block ID 790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-790 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-790 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-790.1709793882.6315835.submit parsl.executors.status_handling INFO: Allocated block ID 791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-791 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-791 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-791.1709793882.632012.submit parsl.executors.status_handling INFO: Allocated block ID 792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-792 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-792 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-792.1709793882.6324441.submit parsl.executors.status_handling INFO: Allocated block ID 793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-793 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-793 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-793.1709793882.6328747.submit parsl.executors.status_handling INFO: Allocated block ID 794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-794 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-794 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-794.1709793882.633299.submit parsl.executors.status_handling INFO: Allocated block ID 795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-795 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-795 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-795.1709793882.6337283.submit parsl.executors.status_handling INFO: Allocated block ID 796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-796 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-796 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-796.1709793882.6341646.submit parsl.executors.status_handling INFO: Allocated block ID 797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-797 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-797 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-797.1709793882.634588.submit parsl.executors.status_handling INFO: Allocated block ID 798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-798 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-798 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-798.1709793882.6350164.submit parsl.executors.status_handling INFO: Allocated block ID 799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-799 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-799 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-799.1709793882.6354394.submit parsl.executors.status_handling INFO: Allocated block ID 800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-800 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-800 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-800.1709793882.635865.submit parsl.executors.status_handling INFO: Allocated block ID 801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-801 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-801 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-801.1709793882.6362987.submit parsl.executors.status_handling INFO: Allocated block ID 802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-802 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-802 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-802.1709793882.6367216.submit parsl.executors.status_handling INFO: Allocated block ID 803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-803 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-803 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-803.1709793882.6371543.submit parsl.executors.status_handling INFO: Allocated block ID 804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-804 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-804 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-804.1709793882.6375773.submit parsl.executors.status_handling INFO: Allocated block ID 805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-805 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-805 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-805.1709793882.638007.submit parsl.executors.status_handling INFO: Allocated block ID 806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-806 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-806 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-806.1709793882.638433.submit parsl.executors.status_handling INFO: Allocated block ID 807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-807 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-807 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-807.1709793882.6388748.submit parsl.executors.status_handling INFO: Allocated block ID 808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-808 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-808 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-808.1709793882.6393068.submit parsl.executors.status_handling INFO: Allocated block ID 809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-809 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-809 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-809.1709793882.6397321.submit parsl.executors.status_handling INFO: Allocated block ID 810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-810 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-810 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-810.1709793882.6402278.submit parsl.executors.status_handling INFO: Allocated block ID 811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-811 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-811 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-811.1709793882.6406586.submit parsl.executors.status_handling INFO: Allocated block ID 812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-812 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-812 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-812.1709793882.6410897.submit parsl.executors.status_handling INFO: Allocated block ID 813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-813 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-813 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-813.1709793882.6415243.submit parsl.executors.status_handling INFO: Allocated block ID 814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-814 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-814 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-814.1709793882.6419575.submit parsl.executors.status_handling INFO: Allocated block ID 815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-815 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-815 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-815.1709793882.6423855.submit parsl.executors.status_handling INFO: Allocated block ID 816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-816 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-816 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-816.1709793882.642817.submit parsl.executors.status_handling INFO: Allocated block ID 817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-817 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-817 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-817.1709793882.643243.submit parsl.executors.status_handling INFO: Allocated block ID 818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-818 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-818 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-818.1709793882.6436648.submit parsl.executors.status_handling INFO: Allocated block ID 819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-819 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-819 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-819.1709793882.6440969.submit parsl.executors.status_handling INFO: Allocated block ID 820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-820 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-820 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-820.1709793882.644518.submit parsl.executors.status_handling INFO: Allocated block ID 821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-821 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-821 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-821.1709793882.6449437.submit parsl.executors.status_handling INFO: Allocated block ID 822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-822 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-822 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-822.1709793882.6453676.submit parsl.executors.status_handling INFO: Allocated block ID 823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-823 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-823 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-823.1709793882.6457937.submit parsl.executors.status_handling INFO: Allocated block ID 824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-824 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-824 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-824.1709793882.6462305.submit parsl.executors.status_handling INFO: Allocated block ID 825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-825 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-825 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-825.1709793882.6466625.submit parsl.executors.status_handling INFO: Allocated block ID 826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-826 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-826 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-826.1709793882.6470985.submit parsl.executors.status_handling INFO: Allocated block ID 827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-827 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-827 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-827.1709793882.6475248.submit parsl.executors.status_handling INFO: Allocated block ID 828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-828 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-828 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-828.1709793882.6479514.submit parsl.executors.status_handling INFO: Allocated block ID 829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-829 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-829 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-829.1709793882.648375.submit parsl.executors.status_handling INFO: Allocated block ID 830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-830 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-830 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-830.1709793882.6487958.submit parsl.executors.status_handling INFO: Allocated block ID 831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-831 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-831 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-831.1709793882.6492357.submit parsl.executors.status_handling INFO: Allocated block ID 832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-832 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-832 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-832.1709793882.6496587.submit parsl.executors.status_handling INFO: Allocated block ID 833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-833 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-833 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-833.1709793882.6610663.submit parsl.executors.status_handling INFO: Allocated block ID 834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-834 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-834 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-834.1709793882.6615584.submit parsl.executors.status_handling INFO: Allocated block ID 835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-835 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-835 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-835.1709793882.6619942.submit parsl.executors.status_handling INFO: Allocated block ID 836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-836 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-836 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-836.1709793882.662423.submit parsl.executors.status_handling INFO: Allocated block ID 837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-837 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-837 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-837.1709793882.6628687.submit parsl.executors.status_handling INFO: Allocated block ID 838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-838 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-838 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-838.1709793882.6632962.submit parsl.executors.status_handling INFO: Allocated block ID 839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-839 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-839 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-839.1709793882.6637182.submit parsl.executors.status_handling INFO: Allocated block ID 840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-840 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-840 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-840.1709793882.664147.submit parsl.executors.status_handling INFO: Allocated block ID 841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-841 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-841 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-841.1709793882.664568.submit parsl.executors.status_handling INFO: Allocated block ID 842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-842 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-842 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-842.1709793882.664995.submit parsl.executors.status_handling INFO: Allocated block ID 843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-843 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-843 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-843.1709793882.6654208.submit parsl.executors.status_handling INFO: Allocated block ID 844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-844 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-844 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-844.1709793882.6658514.submit parsl.executors.status_handling INFO: Allocated block ID 845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-845 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-845 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-845.1709793882.6662734.submit parsl.executors.status_handling INFO: Allocated block ID 846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-846 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-846 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-846.1709793882.6666908.submit parsl.executors.status_handling INFO: Allocated block ID 847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-847 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-847 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-847.1709793882.6671176.submit parsl.executors.status_handling INFO: Allocated block ID 848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-848 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-848 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-848.1709793882.6675375.submit parsl.executors.status_handling INFO: Allocated block ID 849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-849 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-849 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-849.1709793882.6679752.submit parsl.executors.status_handling INFO: Allocated block ID 850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-850 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-850 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-850.1709793882.668401.submit parsl.executors.status_handling INFO: Allocated block ID 851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-851 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-851 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-851.1709793882.6688254.submit parsl.executors.status_handling INFO: Allocated block ID 852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-852 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-852 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-852.1709793882.6692536.submit parsl.executors.status_handling INFO: Allocated block ID 853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-853 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-853 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-853.1709793882.6696796.submit parsl.executors.status_handling INFO: Allocated block ID 854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-854 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-854 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-854.1709793882.670113.submit parsl.executors.status_handling INFO: Allocated block ID 855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-855 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-855 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-855.1709793882.6705399.submit parsl.executors.status_handling INFO: Allocated block ID 856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-856 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-856 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-856.1709793882.682029.submit parsl.executors.status_handling INFO: Allocated block ID 857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-857 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-857 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-857.1709793882.6825361.submit parsl.executors.status_handling INFO: Allocated block ID 858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-858 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-858 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-858.1709793882.6829894.submit parsl.executors.status_handling INFO: Allocated block ID 859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-859 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-859 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-859.1709793882.6834314.submit parsl.executors.status_handling INFO: Allocated block ID 860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-860 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-860 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-860.1709793882.6838756.submit parsl.executors.status_handling INFO: Allocated block ID 861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-861 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-861 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-861.1709793882.6843216.submit parsl.executors.status_handling INFO: Allocated block ID 862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-862 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-862 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-862.1709793882.6847565.submit parsl.executors.status_handling INFO: Allocated block ID 863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-863 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-863 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-863.1709793882.6851993.submit parsl.executors.status_handling INFO: Allocated block ID 864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-864 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-864 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-864.1709793882.6856341.submit parsl.executors.status_handling INFO: Allocated block ID 865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-865 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-865 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-865.1709793882.6860688.submit parsl.executors.status_handling INFO: Allocated block ID 866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-866 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-866 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-866.1709793882.6865041.submit parsl.executors.status_handling INFO: Allocated block ID 867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-867 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-867 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-867.1709793882.6869543.submit parsl.executors.status_handling INFO: Allocated block ID 868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-868 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-868 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-868.1709793882.6873908.submit parsl.executors.status_handling INFO: Allocated block ID 869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-869 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-869 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-869.1709793882.687829.submit parsl.executors.status_handling INFO: Allocated block ID 870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-870 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-870 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-870.1709793882.6882713.submit parsl.executors.status_handling INFO: Allocated block ID 871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-871 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-871 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-871.1709793882.6886914.submit parsl.executors.status_handling INFO: Allocated block ID 872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-872 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-872 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-872.1709793882.7104516.submit parsl.executors.status_handling INFO: Allocated block ID 873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-873 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-873 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-873.1709793882.7110407.submit parsl.executors.status_handling INFO: Allocated block ID 874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-874 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-874 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-874.1709793882.7115817.submit parsl.executors.status_handling INFO: Allocated block ID 875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-875 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-875 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-875.1709793882.712113.submit parsl.executors.status_handling INFO: Allocated block ID 876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-876 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-876 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-876.1709793882.7126358.submit parsl.executors.status_handling INFO: Allocated block ID 877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-877 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-877 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-877.1709793882.7131615.submit parsl.executors.status_handling INFO: Allocated block ID 878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-878 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-878 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-878.1709793882.713682.submit parsl.executors.status_handling INFO: Allocated block ID 879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-879 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-879 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-879.1709793882.7142134.submit parsl.executors.status_handling INFO: Allocated block ID 880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-880 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-880 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-880.1709793882.71474.submit parsl.executors.status_handling INFO: Allocated block ID 881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-881 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-881 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-881.1709793882.7153516.submit parsl.executors.status_handling INFO: Allocated block ID 882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-882 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-882 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-882.1709793882.715881.submit parsl.executors.status_handling INFO: Allocated block ID 883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-883 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-883 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-883.1709793882.7164114.submit parsl.executors.status_handling INFO: Allocated block ID 884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-884 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-884 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-884.1709793882.7169425.submit parsl.executors.status_handling INFO: Allocated block ID 885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-885 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-885 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-885.1709793882.7174764.submit parsl.executors.status_handling INFO: Allocated block ID 886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-886 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-886 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-886.1709793882.71801.submit parsl.executors.status_handling INFO: Allocated block ID 887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-887 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-887 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-887.1709793882.7185354.submit parsl.executors.status_handling INFO: Allocated block ID 888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-888 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-888 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-888.1709793882.7190604.submit parsl.executors.status_handling INFO: Allocated block ID 889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-889 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-889 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-889.1709793882.7195802.submit parsl.executors.status_handling INFO: Allocated block ID 890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-890 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-890 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-890.1709793882.7201111.submit parsl.executors.status_handling INFO: Allocated block ID 891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-891 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-891 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-891.1709793882.7206383.submit parsl.executors.status_handling INFO: Allocated block ID 892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-892 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-892 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-892.1709793882.731394.submit parsl.executors.status_handling INFO: Allocated block ID 893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-893 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-893 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-893.1709793882.7319655.submit parsl.executors.status_handling INFO: Allocated block ID 894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-894 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-894 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-894.1709793882.7324984.submit parsl.executors.status_handling INFO: Allocated block ID 895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-895 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-895 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-895.1709793882.7331433.submit parsl.executors.status_handling INFO: Allocated block ID 896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-896 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-896 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-896.1709793882.7336826.submit parsl.executors.status_handling INFO: Allocated block ID 897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-897 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-897 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-897.1709793882.7342277.submit parsl.executors.status_handling INFO: Allocated block ID 898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-898 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-898 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-898.1709793882.7347531.submit parsl.executors.status_handling INFO: Allocated block ID 899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-899 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-899 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-899.1709793882.7352872.submit parsl.executors.status_handling INFO: Allocated block ID 900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-900 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-900 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-900.1709793882.735816.submit parsl.executors.status_handling INFO: Allocated block ID 901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-901 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-901 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-901.1709793882.7363422.submit parsl.executors.status_handling INFO: Allocated block ID 902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-902 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-902 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-902.1709793882.7368715.submit parsl.executors.status_handling INFO: Allocated block ID 903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-903 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-903 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-903.1709793882.7373965.submit parsl.executors.status_handling INFO: Allocated block ID 904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-904 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-904 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-904.1709793882.737921.submit parsl.executors.status_handling INFO: Allocated block ID 905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-905 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-905 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-905.1709793882.738443.submit parsl.executors.status_handling INFO: Allocated block ID 906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-906 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-906 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-906.1709793882.7389681.submit parsl.executors.status_handling INFO: Allocated block ID 907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-907 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-907 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-907.1709793882.739483.submit parsl.executors.status_handling INFO: Allocated block ID 908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-908 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-908 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-908.1709793882.7431555.submit parsl.executors.status_handling INFO: Allocated block ID 909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-909 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-909 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-909.1709793882.743732.submit parsl.executors.status_handling INFO: Allocated block ID 910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-910 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-910 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-910.1709793882.7442715.submit parsl.executors.status_handling INFO: Allocated block ID 911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-911 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-911 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-911.1709793882.7447917.submit parsl.executors.status_handling INFO: Allocated block ID 912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-912 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-912 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-912.1709793882.7525146.submit parsl.executors.status_handling INFO: Allocated block ID 913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-913 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-913 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-913.1709793882.7530568.submit parsl.executors.status_handling INFO: Allocated block ID 914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-914 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-914 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-914.1709793882.753587.submit parsl.executors.status_handling INFO: Allocated block ID 915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-915 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-915 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-915.1709793882.7541234.submit parsl.executors.status_handling INFO: Allocated block ID 916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-916 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-916 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-916.1709793882.7546527.submit parsl.executors.status_handling INFO: Allocated block ID 917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-917 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-917 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-917.1709793882.7551985.submit parsl.executors.status_handling INFO: Allocated block ID 918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-918 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-918 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-918.1709793882.7557282.submit parsl.executors.status_handling INFO: Allocated block ID 919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-919 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-919 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-919.1709793882.756258.submit parsl.executors.status_handling INFO: Allocated block ID 920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-920 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-920 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-920.1709793882.756779.submit parsl.executors.status_handling INFO: Allocated block ID 921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-921 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-921 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-921.1709793882.7573185.submit parsl.executors.status_handling INFO: Allocated block ID 922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-922 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-922 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-922.1709793882.7669947.submit parsl.executors.status_handling INFO: Allocated block ID 923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-923 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-923 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-923.1709793882.767582.submit parsl.executors.status_handling INFO: Allocated block ID 924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-924 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-924 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-924.1709793882.7681265.submit parsl.executors.status_handling INFO: Allocated block ID 925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-925 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-925 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-925.1709793882.7686522.submit parsl.executors.status_handling INFO: Allocated block ID 926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-926 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-926 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-926.1709793882.7691796.submit parsl.executors.status_handling INFO: Allocated block ID 927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-927 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-927 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-927.1709793882.76971.submit parsl.executors.status_handling INFO: Allocated block ID 928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-928 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-928 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-928.1709793882.770305.submit parsl.executors.status_handling INFO: Allocated block ID 929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-929 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-929 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-929.1709793882.7708395.submit parsl.executors.status_handling INFO: Allocated block ID 930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-930 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-930 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-930.1709793882.7713602.submit parsl.executors.status_handling INFO: Allocated block ID 931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-931 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-931 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-931.1709793882.7718873.submit parsl.executors.status_handling INFO: Allocated block ID 932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-932 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-932 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-932.1709793882.7724068.submit parsl.executors.status_handling INFO: Allocated block ID 933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-933 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-933 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-933.1709793882.7729416.submit parsl.executors.status_handling INFO: Allocated block ID 934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-934 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-934 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-934.1709793882.7734668.submit parsl.executors.status_handling INFO: Allocated block ID 935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-935 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-935 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-935.1709793882.7740517.submit parsl.executors.status_handling INFO: Allocated block ID 936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-936 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-936 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-936.1709793882.7745774.submit parsl.executors.status_handling INFO: Allocated block ID 937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-937 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-937 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-937.1709793882.775107.submit parsl.executors.status_handling INFO: Allocated block ID 938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-938 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-938 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-938.1709793882.7756345.submit parsl.executors.status_handling INFO: Allocated block ID 939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-939 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-939 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-939.1709793882.776181.submit parsl.executors.status_handling INFO: Allocated block ID 940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-940 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-940 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-940.1709793882.7767038.submit parsl.executors.status_handling INFO: Allocated block ID 941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-941 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-941 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-941.1709793882.7772293.submit parsl.executors.status_handling INFO: Allocated block ID 942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-942 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-942 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-942.1709793882.7777505.submit parsl.executors.status_handling INFO: Allocated block ID 943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-943 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-943 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-943.1709793882.7782807.submit parsl.executors.status_handling INFO: Allocated block ID 944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-944 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-944 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-944.1709793882.7788143.submit parsl.executors.status_handling INFO: Allocated block ID 945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-945 2024-03-06 22:44:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-945 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-945.1709793882.7793493.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01fdb637480ff12f6b0e544af7139b9e with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/228df216-055c-4abe-8b4c-530cd2dc9916_biasIsr_1123736_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/228df216-055c-4abe-8b4c-530cd2dc9916_biasIsr_1123736_47.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94c61b55af861d34963685a5eb87ce35 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8c7215cf-eb51-4b3f-b1ba-ce6a064ec660_biasIsr_1123732_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123732/8c7215cf-eb51-4b3f-b1ba-ce6a064ec660_biasIsr_1123732_47.stderr 2024-03-06 22:44:47 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1360 parsl.dataflow.strategy DEBUG: Executor multi has 1360 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-946 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-946 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-946.1709793887.7413058.submit parsl.executors.status_handling INFO: Allocated block ID 947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-947 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-947 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-947.1709793887.7418444.submit parsl.executors.status_handling INFO: Allocated block ID 948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-948 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-948 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-948.1709793887.7422976.submit parsl.executors.status_handling INFO: Allocated block ID 949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-949 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-949 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-949.1709793887.7427397.submit parsl.executors.status_handling INFO: Allocated block ID 950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-950 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-950 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-950.1709793887.7431843.submit parsl.executors.status_handling INFO: Allocated block ID 951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-951 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-951 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-951.1709793887.7436187.submit parsl.executors.status_handling INFO: Allocated block ID 952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-952 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-952 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-952.1709793887.7443314.submit parsl.executors.status_handling INFO: Allocated block ID 953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-953 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-953 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-953.1709793887.7447767.submit parsl.executors.status_handling INFO: Allocated block ID 954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-954 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-954 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-954.1709793887.7452152.submit parsl.executors.status_handling INFO: Allocated block ID 955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-955 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-955 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-955.1709793887.745647.submit parsl.executors.status_handling INFO: Allocated block ID 956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-956 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-956 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-956.1709793887.7460878.submit parsl.executors.status_handling INFO: Allocated block ID 957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-957 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-957 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-957.1709793887.7465265.submit parsl.executors.status_handling INFO: Allocated block ID 958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-958 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-958 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-958.1709793887.74697.submit parsl.executors.status_handling INFO: Allocated block ID 959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-959 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-959 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-959.1709793887.74741.submit parsl.executors.status_handling INFO: Allocated block ID 960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-960 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-960 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-960.1709793887.747884.submit parsl.executors.status_handling INFO: Allocated block ID 961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-961 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-961 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-961.1709793887.7483265.submit parsl.executors.status_handling INFO: Allocated block ID 962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-962 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-962 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-962.1709793887.7487555.submit parsl.executors.status_handling INFO: Allocated block ID 963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-963 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-963 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-963.1709793887.7492256.submit parsl.executors.status_handling INFO: Allocated block ID 964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-964 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-964 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-964.1709793887.7496576.submit parsl.executors.status_handling INFO: Allocated block ID 965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-965 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-965 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-965.1709793887.7500942.submit parsl.executors.status_handling INFO: Allocated block ID 966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-966 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-966 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-966.1709793887.7505195.submit parsl.executors.status_handling INFO: Allocated block ID 967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-967 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-967 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-967.1709793887.7509472.submit parsl.executors.status_handling INFO: Allocated block ID 968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-968 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-968 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-968.1709793887.7513757.submit parsl.executors.status_handling INFO: Allocated block ID 969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-969 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-969 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-969.1709793887.7518046.submit parsl.executors.status_handling INFO: Allocated block ID 970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-970 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-970 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-970.1709793887.752244.submit parsl.executors.status_handling INFO: Allocated block ID 971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-971 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-971 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-971.1709793887.752678.submit parsl.executors.status_handling INFO: Allocated block ID 972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-972 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-972 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-972.1709793887.7531161.submit parsl.executors.status_handling INFO: Allocated block ID 973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-973 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-973 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-973.1709793887.753545.submit parsl.executors.status_handling INFO: Allocated block ID 974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-974 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-974 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-974.1709793887.7540047.submit parsl.executors.status_handling INFO: Allocated block ID 975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-975 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-975 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-975.1709793887.7544274.submit parsl.executors.status_handling INFO: Allocated block ID 976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-976 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-976 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-976.1709793887.7548573.submit parsl.executors.status_handling INFO: Allocated block ID 977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-977 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-977 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-977.1709793887.7552812.submit parsl.executors.status_handling INFO: Allocated block ID 978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-978 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-978 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-978.1709793887.7557065.submit parsl.executors.status_handling INFO: Allocated block ID 979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-979 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-979 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-979.1709793887.756144.submit parsl.executors.status_handling INFO: Allocated block ID 980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-980 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-980 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-980.1709793887.756565.submit parsl.executors.status_handling INFO: Allocated block ID 981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-981 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-981 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-981.1709793887.7569942.submit parsl.executors.status_handling INFO: Allocated block ID 982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-982 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-982 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-982.1709793887.757419.submit parsl.executors.status_handling INFO: Allocated block ID 983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-983 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-983 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-983.1709793887.7578533.submit parsl.executors.status_handling INFO: Allocated block ID 984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-984 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-984 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-984.1709793887.7582777.submit parsl.executors.status_handling INFO: Allocated block ID 985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-985 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-985 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-985.1709793887.7587056.submit parsl.executors.status_handling INFO: Allocated block ID 986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-986 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-986 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-986.1709793887.7591343.submit parsl.executors.status_handling INFO: Allocated block ID 987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-987 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-987 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-987.1709793887.75956.submit parsl.executors.status_handling INFO: Allocated block ID 988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-988 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-988 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-988.1709793887.7600248.submit parsl.executors.status_handling INFO: Allocated block ID 989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-989 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-989 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-989.1709793887.7604592.submit parsl.executors.status_handling INFO: Allocated block ID 990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-990 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-990 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-990.1709793887.7608922.submit parsl.executors.status_handling INFO: Allocated block ID 991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-991 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-991 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-991.1709793887.7613194.submit parsl.executors.status_handling INFO: Allocated block ID 992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-992 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-992 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-992.1709793887.761742.submit parsl.executors.status_handling INFO: Allocated block ID 993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-993 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-993 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-993.1709793887.7621708.submit parsl.executors.status_handling INFO: Allocated block ID 994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-994 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-994 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-994.1709793887.7625992.submit parsl.executors.status_handling INFO: Allocated block ID 995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-995 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-995 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-995.1709793887.763036.submit parsl.executors.status_handling INFO: Allocated block ID 996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-996 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-996 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-996.1709793887.7634652.submit parsl.executors.status_handling INFO: Allocated block ID 997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-997 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-997 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-997.1709793887.7639012.submit parsl.executors.status_handling INFO: Allocated block ID 998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-998 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-998 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-998.1709793887.7643244.submit parsl.executors.status_handling INFO: Allocated block ID 999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-999 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-999 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-999.1709793887.7647488.submit parsl.executors.status_handling INFO: Allocated block ID 1000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1000 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1000 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1000.1709793887.7651842.submit parsl.executors.status_handling INFO: Allocated block ID 1001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1001 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1001 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1001.1709793887.765607.submit parsl.executors.status_handling INFO: Allocated block ID 1002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1002 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1002 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1002.1709793887.766034.submit parsl.executors.status_handling INFO: Allocated block ID 1003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1003 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1003 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1003.1709793887.7664592.submit parsl.executors.status_handling INFO: Allocated block ID 1004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1004 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1004 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1004.1709793887.7669053.submit parsl.executors.status_handling INFO: Allocated block ID 1005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1005 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1005 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1005.1709793887.7673273.submit parsl.executors.status_handling INFO: Allocated block ID 1006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1006 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1006 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1006.1709793887.767755.submit parsl.executors.status_handling INFO: Allocated block ID 1007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1007 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1007 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1007.1709793887.7681916.submit parsl.executors.status_handling INFO: Allocated block ID 1008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1008 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1008 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1008.1709793887.7686133.submit parsl.executors.status_handling INFO: Allocated block ID 1009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1009 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1009 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1009.1709793887.7690463.submit parsl.executors.status_handling INFO: Allocated block ID 1010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1010 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1010 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1010.1709793887.769473.submit parsl.executors.status_handling INFO: Allocated block ID 1011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1011 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1011 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1011.1709793887.769909.submit parsl.executors.status_handling INFO: Allocated block ID 1012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1012 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1012 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1012.1709793887.7703373.submit parsl.executors.status_handling INFO: Allocated block ID 1013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1013 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1013 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1013.1709793887.7707586.submit parsl.executors.status_handling INFO: Allocated block ID 1014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1014 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1014 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1014.1709793887.7712233.submit parsl.executors.status_handling INFO: Allocated block ID 1015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1015 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1015 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1015.1709793887.7716477.submit parsl.executors.status_handling INFO: Allocated block ID 1016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1016 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1016 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1016.1709793887.7720816.submit parsl.executors.status_handling INFO: Allocated block ID 1017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1017 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1017 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1017.1709793887.7725093.submit parsl.executors.status_handling INFO: Allocated block ID 1018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1018 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1018 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1018.1709793887.7729404.submit parsl.executors.status_handling INFO: Allocated block ID 1019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1019 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1019 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1019.1709793887.7733717.submit parsl.executors.status_handling INFO: Allocated block ID 1020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1020 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1020 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1020.1709793887.7737968.submit parsl.executors.status_handling INFO: Allocated block ID 1021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1021 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1021 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1021.1709793887.7742307.submit parsl.executors.status_handling INFO: Allocated block ID 1022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1022 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1022 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1022.1709793887.774658.submit parsl.executors.status_handling INFO: Allocated block ID 1023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1023 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1023 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1023.1709793887.7750895.submit parsl.executors.status_handling INFO: Allocated block ID 1024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1024 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1024 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1024.1709793887.7755165.submit parsl.executors.status_handling INFO: Allocated block ID 1025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1025 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1025 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1025.1709793887.7759502.submit parsl.executors.status_handling INFO: Allocated block ID 1026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1026 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1026 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1026.1709793887.776374.submit parsl.executors.status_handling INFO: Allocated block ID 1027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1027 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1027 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1027.1709793887.776796.submit parsl.executors.status_handling INFO: Allocated block ID 1028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1028 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1028 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1028.1709793887.7772236.submit parsl.executors.status_handling INFO: Allocated block ID 1029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1029 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1029 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1029.1709793887.7776725.submit parsl.executors.status_handling INFO: Allocated block ID 1030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1030 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1030 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1030.1709793887.7781193.submit parsl.executors.status_handling INFO: Allocated block ID 1031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1031 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1031 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1031.1709793887.7785625.submit parsl.executors.status_handling INFO: Allocated block ID 1032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1032 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1032 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1032.1709793887.7790003.submit parsl.executors.status_handling INFO: Allocated block ID 1033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1033 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1033 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1033.1709793887.779429.submit parsl.executors.status_handling INFO: Allocated block ID 1034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1034 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1034 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1034.1709793887.7798605.submit parsl.executors.status_handling INFO: Allocated block ID 1035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1035 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1035 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1035.1709793887.7802856.submit parsl.executors.status_handling INFO: Allocated block ID 1036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1036 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1036 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1036.1709793887.7807128.submit parsl.executors.status_handling INFO: Allocated block ID 1037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1037 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1037 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1037.1709793887.7811584.submit parsl.executors.status_handling INFO: Allocated block ID 1038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1038 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1038 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1038.1709793887.7815838.submit parsl.executors.status_handling INFO: Allocated block ID 1039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1039 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1039 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1039.1709793887.7820444.submit parsl.executors.status_handling INFO: Allocated block ID 1040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1040 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1040 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1040.1709793887.7824757.submit parsl.executors.status_handling INFO: Allocated block ID 1041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1041 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1041 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1041.1709793887.7829084.submit parsl.executors.status_handling INFO: Allocated block ID 1042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1042 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1042 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1042.1709793887.7833364.submit parsl.executors.status_handling INFO: Allocated block ID 1043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1043 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1043 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1043.1709793887.7837722.submit parsl.executors.status_handling INFO: Allocated block ID 1044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1044 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1044 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1044.1709793887.7842145.submit parsl.executors.status_handling INFO: Allocated block ID 1045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1045 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1045 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1045.1709793887.784637.submit parsl.executors.status_handling INFO: Allocated block ID 1046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1046 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1046 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1046.1709793887.785066.submit parsl.executors.status_handling INFO: Allocated block ID 1047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1047 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1047 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1047.1709793887.785492.submit parsl.executors.status_handling INFO: Allocated block ID 1048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1048 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1048 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1048.1709793887.7859213.submit parsl.executors.status_handling INFO: Allocated block ID 1049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1049 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1049 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1049.1709793887.7863457.submit parsl.executors.status_handling INFO: Allocated block ID 1050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1050 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1050 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1050.1709793887.7867658.submit parsl.executors.status_handling INFO: Allocated block ID 1051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1051 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1051 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1051.1709793887.7871969.submit parsl.executors.status_handling INFO: Allocated block ID 1052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1052 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1052 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1052.1709793887.7876213.submit parsl.executors.status_handling INFO: Allocated block ID 1053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1053 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1053 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1053.1709793887.7880569.submit parsl.executors.status_handling INFO: Allocated block ID 1054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1054 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1054 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1054.1709793887.7884843.submit parsl.executors.status_handling INFO: Allocated block ID 1055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1055 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1055 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1055.1709793887.7889175.submit parsl.executors.status_handling INFO: Allocated block ID 1056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1056 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1056 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1056.1709793887.7893453.submit parsl.executors.status_handling INFO: Allocated block ID 1057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1057 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1057 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1057.1709793887.7897694.submit parsl.executors.status_handling INFO: Allocated block ID 1058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1058 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1058 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1058.1709793887.790197.submit parsl.executors.status_handling INFO: Allocated block ID 1059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1059 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1059 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1059.1709793887.7906177.submit parsl.executors.status_handling INFO: Allocated block ID 1060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1060 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1060 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1060.1709793887.7910454.submit parsl.executors.status_handling INFO: Allocated block ID 1061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1061 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1061 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1061.1709793887.7914667.submit parsl.executors.status_handling INFO: Allocated block ID 1062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1062 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1062 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1062.1709793887.7918994.submit parsl.executors.status_handling INFO: Allocated block ID 1063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1063 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1063 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1063.1709793887.792325.submit parsl.executors.status_handling INFO: Allocated block ID 1064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1064 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1064 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1064.1709793887.7927513.submit parsl.executors.status_handling INFO: Allocated block ID 1065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1065 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1065 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1065.1709793887.793218.submit parsl.executors.status_handling INFO: Allocated block ID 1066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1066 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1066 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1066.1709793887.7936475.submit parsl.executors.status_handling INFO: Allocated block ID 1067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1067 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1067 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1067.1709793887.7940824.submit parsl.executors.status_handling INFO: Allocated block ID 1068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1068 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1068 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1068.1709793887.7945113.submit parsl.executors.status_handling INFO: Allocated block ID 1069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1069 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1069 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1069.1709793887.7949443.submit parsl.executors.status_handling INFO: Allocated block ID 1070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1070 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1070 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1070.1709793887.7953675.submit parsl.executors.status_handling INFO: Allocated block ID 1071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1071 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1071 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1071.1709793887.7957962.submit parsl.executors.status_handling INFO: Allocated block ID 1072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1072 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1072 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1072.1709793887.7962434.submit parsl.executors.status_handling INFO: Allocated block ID 1073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1073 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1073 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1073.1709793887.7966712.submit parsl.executors.status_handling INFO: Allocated block ID 1074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1074 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1074 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1074.1709793887.7971008.submit parsl.executors.status_handling INFO: Allocated block ID 1075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1075 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1075 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1075.1709793887.7975295.submit parsl.executors.status_handling INFO: Allocated block ID 1076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1076 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1076 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1076.1709793887.797958.submit parsl.executors.status_handling INFO: Allocated block ID 1077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1077 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1077 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1077.1709793887.798392.submit parsl.executors.status_handling INFO: Allocated block ID 1078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1078 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1078 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1078.1709793887.798828.submit parsl.executors.status_handling INFO: Allocated block ID 1079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1079 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1079 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1079.1709793887.7992625.submit parsl.executors.status_handling INFO: Allocated block ID 1080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1080 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1080 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1080.1709793887.7996929.submit parsl.executors.status_handling INFO: Allocated block ID 1081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1081 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1081 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1081.1709793887.800129.submit parsl.executors.status_handling INFO: Allocated block ID 1082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1082 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1082 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1082.1709793887.8005517.submit parsl.executors.status_handling INFO: Allocated block ID 1083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1083 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1083 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1083.1709793887.8009825.submit parsl.executors.status_handling INFO: Allocated block ID 1084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1084 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1084 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1084.1709793887.8014107.submit parsl.executors.status_handling INFO: Allocated block ID 1085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1085 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1085 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1085.1709793887.8018413.submit parsl.executors.status_handling INFO: Allocated block ID 1086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1086 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1086 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1086.1709793887.802262.submit parsl.executors.status_handling INFO: Allocated block ID 1087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1087 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1087 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1087.1709793887.8026886.submit parsl.executors.status_handling INFO: Allocated block ID 1088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1088 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1088 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1088.1709793887.8031278.submit parsl.executors.status_handling INFO: Allocated block ID 1089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1089 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1089 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1089.1709793887.8035598.submit parsl.executors.status_handling INFO: Allocated block ID 1090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1090 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1090 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1090.1709793887.8040314.submit parsl.executors.status_handling INFO: Allocated block ID 1091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1091 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1091 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1091.1709793887.8044772.submit parsl.executors.status_handling INFO: Allocated block ID 1092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1092 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1092 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1092.1709793887.8049147.submit parsl.executors.status_handling INFO: Allocated block ID 1093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1093 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1093 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1093.1709793887.805362.submit parsl.executors.status_handling INFO: Allocated block ID 1094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1094 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1094 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1094.1709793887.8058007.submit parsl.executors.status_handling INFO: Allocated block ID 1095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1095 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1095 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1095.1709793887.8062441.submit parsl.executors.status_handling INFO: Allocated block ID 1096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1096 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1096 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1096.1709793887.8066878.submit parsl.executors.status_handling INFO: Allocated block ID 1097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1097 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1097 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1097.1709793887.8071208.submit parsl.executors.status_handling INFO: Allocated block ID 1098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1098 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1098 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1098.1709793887.8075466.submit parsl.executors.status_handling INFO: Allocated block ID 1099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1099 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1099 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1099.1709793887.8080018.submit parsl.executors.status_handling INFO: Allocated block ID 1100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1100 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1100 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1100.1709793887.808442.submit parsl.executors.status_handling INFO: Allocated block ID 1101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1101 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1101 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1101.1709793887.8088896.submit parsl.executors.status_handling INFO: Allocated block ID 1102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1102 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1102 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1102.1709793887.809331.submit parsl.executors.status_handling INFO: Allocated block ID 1103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1103 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1103 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1103.1709793887.8097627.submit parsl.executors.status_handling INFO: Allocated block ID 1104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1104 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1104 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1104.1709793887.8101964.submit parsl.executors.status_handling INFO: Allocated block ID 1105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1105 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1105 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1105.1709793887.8106203.submit parsl.executors.status_handling INFO: Allocated block ID 1106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1106 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1106 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1106.1709793887.8110542.submit parsl.executors.status_handling INFO: Allocated block ID 1107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1107 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1107 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1107.1709793887.8114908.submit parsl.executors.status_handling INFO: Allocated block ID 1108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1108 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1108 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1108.1709793887.8119514.submit parsl.executors.status_handling INFO: Allocated block ID 1109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1109 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1109 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1109.1709793887.8123949.submit parsl.executors.status_handling INFO: Allocated block ID 1110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1110 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1110 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1110.1709793887.8128405.submit parsl.executors.status_handling INFO: Allocated block ID 1111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1111 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1111 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1111.1709793887.813283.submit parsl.executors.status_handling INFO: Allocated block ID 1112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1112 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1112 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1112.1709793887.8137074.submit parsl.executors.status_handling INFO: Allocated block ID 1113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1113 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1113 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1113.1709793887.8141499.submit parsl.executors.status_handling INFO: Allocated block ID 1114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1114 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1114 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1114.1709793887.814585.submit parsl.executors.status_handling INFO: Allocated block ID 1115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1115 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1115 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1115.1709793887.815083.submit parsl.executors.status_handling INFO: Allocated block ID 1116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1116 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1116 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1116.1709793887.8155317.submit parsl.executors.status_handling INFO: Allocated block ID 1117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1117 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1117 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1117.1709793887.815979.submit parsl.executors.status_handling INFO: Allocated block ID 1118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1118 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1118 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1118.1709793887.816418.submit parsl.executors.status_handling INFO: Allocated block ID 1119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1119 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1119 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1119.1709793887.8168657.submit parsl.executors.status_handling INFO: Allocated block ID 1120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1120 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1120 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1120.1709793887.8173122.submit parsl.executors.status_handling INFO: Allocated block ID 1121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1121 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1121 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1121.1709793887.8177469.submit parsl.executors.status_handling INFO: Allocated block ID 1122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1122 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1122 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1122.1709793887.818194.submit parsl.executors.status_handling INFO: Allocated block ID 1123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1123 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1123 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1123.1709793887.8186185.submit parsl.executors.status_handling INFO: Allocated block ID 1124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1124 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1124 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1124.1709793887.8190522.submit parsl.executors.status_handling INFO: Allocated block ID 1125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1125 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1125 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1125.1709793887.8194768.submit parsl.executors.status_handling INFO: Allocated block ID 1126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1126 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1126 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1126.1709793887.81991.submit parsl.executors.status_handling INFO: Allocated block ID 1127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1127 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1127 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1127.1709793887.8203442.submit parsl.executors.status_handling INFO: Allocated block ID 1128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1128 2024-03-06 22:44:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1128 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1128.1709793887.820776.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1360 parsl.dataflow.strategy DEBUG: Executor multi has 1360 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 1129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1129 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1129 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1129.1709793892.6255.submit parsl.executors.status_handling INFO: Allocated block ID 1130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1130 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1130 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1130.1709793892.6260097.submit parsl.executors.status_handling INFO: Allocated block ID 1131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1131 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1131 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1131.1709793892.6264546.submit parsl.executors.status_handling INFO: Allocated block ID 1132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1132 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1132 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1132.1709793892.6268923.submit parsl.executors.status_handling INFO: Allocated block ID 1133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1133 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1133 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1133.1709793892.627321.submit parsl.executors.status_handling INFO: Allocated block ID 1134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1134 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1134 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1134.1709793892.6277547.submit parsl.executors.status_handling INFO: Allocated block ID 1135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1135 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1135 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1135.1709793892.6281917.submit parsl.executors.status_handling INFO: Allocated block ID 1136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1136 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1136 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1136.1709793892.628621.submit parsl.executors.status_handling INFO: Allocated block ID 1137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1137 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1137 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1137.1709793892.6290615.submit parsl.executors.status_handling INFO: Allocated block ID 1138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1138 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1138 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1138.1709793892.6294959.submit parsl.executors.status_handling INFO: Allocated block ID 1139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1139 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1139 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1139.1709793892.6299329.submit parsl.executors.status_handling INFO: Allocated block ID 1140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1140 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1140 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1140.1709793892.6303618.submit parsl.executors.status_handling INFO: Allocated block ID 1141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1141 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1141 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1141.1709793892.630789.submit parsl.executors.status_handling INFO: Allocated block ID 1142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1142 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1142 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1142.1709793892.6312287.submit parsl.executors.status_handling INFO: Allocated block ID 1143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1143 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1143 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1143.1709793892.631653.submit parsl.executors.status_handling INFO: Allocated block ID 1144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1144 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1144 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1144.1709793892.6320848.submit parsl.executors.status_handling INFO: Allocated block ID 1145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1145 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1145 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1145.1709793892.6325154.submit parsl.executors.status_handling INFO: Allocated block ID 1146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1146 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1146 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1146.1709793892.6329463.submit parsl.executors.status_handling INFO: Allocated block ID 1147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1147 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1147 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1147.1709793892.633371.submit parsl.executors.status_handling INFO: Allocated block ID 1148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1148 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1148 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1148.1709793892.6338003.submit parsl.executors.status_handling INFO: Allocated block ID 1149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1149 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1149 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1149.1709793892.634249.submit parsl.executors.status_handling INFO: Allocated block ID 1150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1150 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1150 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1150.1709793892.634683.submit parsl.executors.status_handling INFO: Allocated block ID 1151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1151 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1151 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1151.1709793892.6351228.submit parsl.executors.status_handling INFO: Allocated block ID 1152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1152 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1152 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1152.1709793892.6355548.submit parsl.executors.status_handling INFO: Allocated block ID 1153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1153 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1153 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1153.1709793892.6461573.submit parsl.executors.status_handling INFO: Allocated block ID 1154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1154 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1154 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1154.1709793892.6466212.submit parsl.executors.status_handling INFO: Allocated block ID 1155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1155 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1155 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1155.1709793892.6470683.submit parsl.executors.status_handling INFO: Allocated block ID 1156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1156 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1156 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1156.1709793892.6474972.submit parsl.executors.status_handling INFO: Allocated block ID 1157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1157 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1157 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1157.1709793892.647928.submit parsl.executors.status_handling INFO: Allocated block ID 1158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1158 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1158 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1158.1709793892.648353.submit parsl.executors.status_handling INFO: Allocated block ID 1159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1159 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1159 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1159.1709793892.6487777.submit parsl.executors.status_handling INFO: Allocated block ID 1160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1160 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1160 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1160.1709793892.6492147.submit parsl.executors.status_handling INFO: Allocated block ID 1161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1161 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1161 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1161.1709793892.6496484.submit parsl.executors.status_handling INFO: Allocated block ID 1162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1162 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1162 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1162.1709793892.6500828.submit parsl.executors.status_handling INFO: Allocated block ID 1163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1163 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1163 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1163.1709793892.6505113.submit parsl.executors.status_handling INFO: Allocated block ID 1164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1164 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1164 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1164.1709793892.650944.submit parsl.executors.status_handling INFO: Allocated block ID 1165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1165 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1165 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1165.1709793892.651368.submit parsl.executors.status_handling INFO: Allocated block ID 1166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1166 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1166 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1166.1709793892.651792.submit parsl.executors.status_handling INFO: Allocated block ID 1167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1167 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1167 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1167.1709793892.6522295.submit parsl.executors.status_handling INFO: Allocated block ID 1168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1168 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1168 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1168.1709793892.6526544.submit parsl.executors.status_handling INFO: Allocated block ID 1169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1169 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1169 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1169.1709793892.6530845.submit parsl.executors.status_handling INFO: Allocated block ID 1170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1170 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1170 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1170.1709793892.6535056.submit parsl.executors.status_handling INFO: Allocated block ID 1171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1171 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1171 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1171.1709793892.6539352.submit parsl.executors.status_handling INFO: Allocated block ID 1172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1172 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1172 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1172.1709793892.6543634.submit parsl.executors.status_handling INFO: Allocated block ID 1173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1173 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1173 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1173.1709793892.6547942.submit parsl.executors.status_handling INFO: Allocated block ID 1174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1174 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1174 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1174.1709793892.6654944.submit parsl.executors.status_handling INFO: Allocated block ID 1175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1175 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1175 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1175.1709793892.6659489.submit parsl.executors.status_handling INFO: Allocated block ID 1176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1176 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1176 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1176.1709793892.6663923.submit parsl.executors.status_handling INFO: Allocated block ID 1177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1177 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1177 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1177.1709793892.6668234.submit parsl.executors.status_handling INFO: Allocated block ID 1178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1178 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1178 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1178.1709793892.6672528.submit parsl.executors.status_handling INFO: Allocated block ID 1179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1179 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1179 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1179.1709793892.6676807.submit parsl.executors.status_handling INFO: Allocated block ID 1180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1180 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1180 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1180.1709793892.668115.submit parsl.executors.status_handling INFO: Allocated block ID 1181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1181 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1181 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1181.1709793892.6685417.submit parsl.executors.status_handling INFO: Allocated block ID 1182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1182 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1182 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1182.1709793892.6689723.submit parsl.executors.status_handling INFO: Allocated block ID 1183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1183 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1183 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1183.1709793892.6694033.submit parsl.executors.status_handling INFO: Allocated block ID 1184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1184 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1184 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1184.1709793892.6698396.submit parsl.executors.status_handling INFO: Allocated block ID 1185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1185 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1185 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1185.1709793892.670276.submit parsl.executors.status_handling INFO: Allocated block ID 1186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1186 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1186 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1186.1709793892.6707017.submit parsl.executors.status_handling INFO: Allocated block ID 1187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1187 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1187 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1187.1709793892.671134.submit parsl.executors.status_handling INFO: Allocated block ID 1188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1188 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1188 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1188.1709793892.6715555.submit parsl.executors.status_handling INFO: Allocated block ID 1189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1189 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1189 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1189.1709793892.6719875.submit parsl.executors.status_handling INFO: Allocated block ID 1190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1190 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1190 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1190.1709793892.6724215.submit parsl.executors.status_handling INFO: Allocated block ID 1191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1191 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1191 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1191.1709793892.6728559.submit parsl.executors.status_handling INFO: Allocated block ID 1192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1192 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1192 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1192.1709793892.6732793.submit parsl.executors.status_handling INFO: Allocated block ID 1193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1193 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1193 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1193.1709793892.6737015.submit parsl.executors.status_handling INFO: Allocated block ID 1194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1194 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1194 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1194.1709793892.684316.submit parsl.executors.status_handling INFO: Allocated block ID 1195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1195 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1195 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1195.1709793892.68477.submit parsl.executors.status_handling INFO: Allocated block ID 1196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1196 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1196 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1196.1709793892.6852357.submit parsl.executors.status_handling INFO: Allocated block ID 1197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1197 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1197 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1197.1709793892.685691.submit parsl.executors.status_handling INFO: Allocated block ID 1198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1198 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1198 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1198.1709793892.6861515.submit parsl.executors.status_handling INFO: Allocated block ID 1199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1199 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1199 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1199.1709793892.686596.submit parsl.executors.status_handling INFO: Allocated block ID 1200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1200 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1200 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1200.1709793892.6870437.submit parsl.executors.status_handling INFO: Allocated block ID 1201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1201 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1201 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1201.1709793892.687481.submit parsl.executors.status_handling INFO: Allocated block ID 1202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1202 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1202 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1202.1709793892.68793.submit parsl.executors.status_handling INFO: Allocated block ID 1203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1203 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1203 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1203.1709793892.6883712.submit parsl.executors.status_handling INFO: Allocated block ID 1204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1204 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1204 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1204.1709793892.688812.submit parsl.executors.status_handling INFO: Allocated block ID 1205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1205 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1205 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1205.1709793892.6892495.submit parsl.executors.status_handling INFO: Allocated block ID 1206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1206 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1206 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1206.1709793892.6896875.submit parsl.executors.status_handling INFO: Allocated block ID 1207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1207 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1207 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1207.1709793892.6901388.submit parsl.executors.status_handling INFO: Allocated block ID 1208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1208 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1208 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1208.1709793892.6905675.submit parsl.executors.status_handling INFO: Allocated block ID 1209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1209 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1209 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1209.1709793892.691005.submit parsl.executors.status_handling INFO: Allocated block ID 1210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1210 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1210 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1210.1709793892.691433.submit parsl.executors.status_handling INFO: Allocated block ID 1211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1211 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1211 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1211.1709793892.691867.submit parsl.executors.status_handling INFO: Allocated block ID 1212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1212 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1212 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1212.1709793892.6922972.submit parsl.executors.status_handling INFO: Allocated block ID 1213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1213 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1213 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1213.1709793892.6927238.submit parsl.executors.status_handling INFO: Allocated block ID 1214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1214 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1214 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1214.1709793892.7034245.submit parsl.executors.status_handling INFO: Allocated block ID 1215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1215 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1215 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1215.1709793892.7038743.submit parsl.executors.status_handling INFO: Allocated block ID 1216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1216 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1216 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1216.1709793892.704317.submit parsl.executors.status_handling INFO: Allocated block ID 1217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1217 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1217 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1217.1709793892.704749.submit parsl.executors.status_handling INFO: Allocated block ID 1218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1218 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1218 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1218.1709793892.7051878.submit parsl.executors.status_handling INFO: Allocated block ID 1219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1219 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1219 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1219.1709793892.705621.submit parsl.executors.status_handling INFO: Allocated block ID 1220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1220 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1220 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1220.1709793892.7060554.submit parsl.executors.status_handling INFO: Allocated block ID 1221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1221 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1221 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1221.1709793892.706492.submit parsl.executors.status_handling INFO: Allocated block ID 1222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1222 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1222 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1222.1709793892.7069285.submit parsl.executors.status_handling INFO: Allocated block ID 1223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1223 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1223 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1223.1709793892.7073593.submit parsl.executors.status_handling INFO: Allocated block ID 1224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1224 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1224 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1224.1709793892.7077842.submit parsl.executors.status_handling INFO: Allocated block ID 1225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1225 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1225 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1225.1709793892.7082193.submit parsl.executors.status_handling INFO: Allocated block ID 1226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1226 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1226 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1226.1709793892.7086494.submit parsl.executors.status_handling INFO: Allocated block ID 1227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1227 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1227 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1227.1709793892.7090871.submit parsl.executors.status_handling INFO: Allocated block ID 1228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1228 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1228 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1228.1709793892.7095127.submit parsl.executors.status_handling INFO: Allocated block ID 1229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1229 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1229 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1229.1709793892.7099452.submit parsl.executors.status_handling INFO: Allocated block ID 1230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1230 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1230 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1230.1709793892.7103674.submit parsl.executors.status_handling INFO: Allocated block ID 1231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1231 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1231 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1231.1709793892.7107944.submit parsl.executors.status_handling INFO: Allocated block ID 1232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1232 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1232 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1232.1709793892.711231.submit parsl.executors.status_handling INFO: Allocated block ID 1233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1233 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1233 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1233.1709793892.7116628.submit parsl.executors.status_handling INFO: Allocated block ID 1234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1234 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1234 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1234.1709793892.7222617.submit parsl.executors.status_handling INFO: Allocated block ID 1235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1235 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1235 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1235.1709793892.7227097.submit parsl.executors.status_handling INFO: Allocated block ID 1236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1236 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1236 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1236.1709793892.7231536.submit parsl.executors.status_handling INFO: Allocated block ID 1237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1237 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1237 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1237.1709793892.723595.submit parsl.executors.status_handling INFO: Allocated block ID 1238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1238 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1238 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1238.1709793892.7240548.submit parsl.executors.status_handling INFO: Allocated block ID 1239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1239 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1239 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1239.1709793892.7244935.submit parsl.executors.status_handling INFO: Allocated block ID 1240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1240 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1240 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1240.1709793892.7249463.submit parsl.executors.status_handling INFO: Allocated block ID 1241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1241 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1241 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1241.1709793892.7253864.submit parsl.executors.status_handling INFO: Allocated block ID 1242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1242 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1242 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1242.1709793892.7258177.submit parsl.executors.status_handling INFO: Allocated block ID 1243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1243 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1243 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1243.1709793892.7262764.submit parsl.executors.status_handling INFO: Allocated block ID 1244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1244 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1244 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1244.1709793892.7267177.submit parsl.executors.status_handling INFO: Allocated block ID 1245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1245 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1245 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1245.1709793892.7271783.submit parsl.executors.status_handling INFO: Allocated block ID 1246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1246 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1246 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1246.1709793892.7276046.submit parsl.executors.status_handling INFO: Allocated block ID 1247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1247 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1247 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1247.1709793892.7280352.submit parsl.executors.status_handling INFO: Allocated block ID 1248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1248 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1248 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1248.1709793892.7284636.submit parsl.executors.status_handling INFO: Allocated block ID 1249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1249 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1249 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1249.1709793892.7288952.submit parsl.executors.status_handling INFO: Allocated block ID 1250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1250 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1250 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1250.1709793892.7293236.submit parsl.executors.status_handling INFO: Allocated block ID 1251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1251 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1251 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1251.1709793892.729749.submit parsl.executors.status_handling INFO: Allocated block ID 1252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1252 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1252 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1252.1709793892.7301822.submit parsl.executors.status_handling INFO: Allocated block ID 1253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1253 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1253 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1253.1709793892.73061.submit parsl.executors.status_handling INFO: Allocated block ID 1254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1254 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1254 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1254.1709793892.7353704.submit parsl.executors.status_handling INFO: Allocated block ID 1255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1255 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1255 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1255.1709793892.7358322.submit parsl.executors.status_handling INFO: Allocated block ID 1256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1256 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1256 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1256.1709793892.7362702.submit parsl.executors.status_handling INFO: Allocated block ID 1257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1257 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1257 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1257.1709793892.7367127.submit parsl.executors.status_handling INFO: Allocated block ID 1258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1258 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1258 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1258.1709793892.7371752.submit parsl.executors.status_handling INFO: Allocated block ID 1259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1259 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1259 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1259.1709793892.7376177.submit parsl.executors.status_handling INFO: Allocated block ID 1260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1260 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1260 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1260.1709793892.7380764.submit parsl.executors.status_handling INFO: Allocated block ID 1261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1261 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1261 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1261.1709793892.7385116.submit parsl.executors.status_handling INFO: Allocated block ID 1262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1262 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1262 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1262.1709793892.7389512.submit parsl.executors.status_handling INFO: Allocated block ID 1263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1263 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1263 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1263.1709793892.7393808.submit parsl.executors.status_handling INFO: Allocated block ID 1264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1264 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1264 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1264.1709793892.7398036.submit parsl.executors.status_handling INFO: Allocated block ID 1265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1265 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1265 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1265.1709793892.7402341.submit parsl.executors.status_handling INFO: Allocated block ID 1266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1266 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1266 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1266.1709793892.7406576.submit parsl.executors.status_handling INFO: Allocated block ID 1267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1267 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1267 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1267.1709793892.741093.submit parsl.executors.status_handling INFO: Allocated block ID 1268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1268 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1268 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1268.1709793892.7415233.submit parsl.executors.status_handling INFO: Allocated block ID 1269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1269 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1269 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1269.1709793892.7419646.submit parsl.executors.status_handling INFO: Allocated block ID 1270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1270 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1270 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1270.1709793892.742401.submit parsl.executors.status_handling INFO: Allocated block ID 1271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1271 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1271 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1271.1709793892.7428339.submit parsl.executors.status_handling INFO: Allocated block ID 1272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1272 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1272 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1272.1709793892.7432668.submit parsl.executors.status_handling INFO: Allocated block ID 1273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1273 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1273 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1273.1709793892.743696.submit parsl.executors.status_handling INFO: Allocated block ID 1274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1274 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1274 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1274.1709793892.744137.submit parsl.executors.status_handling INFO: Allocated block ID 1275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1275 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1275 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1275.1709793892.7445643.submit parsl.executors.status_handling INFO: Allocated block ID 1276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1276 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1276 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1276.1709793892.7449994.submit parsl.executors.status_handling INFO: Allocated block ID 1277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1277 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1277 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1277.1709793892.7454295.submit parsl.executors.status_handling INFO: Allocated block ID 1278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1278 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1278 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1278.1709793892.745974.submit parsl.executors.status_handling INFO: Allocated block ID 1279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1279 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1279 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1279.1709793892.746427.submit parsl.executors.status_handling INFO: Allocated block ID 1280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1280 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1280 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1280.1709793892.7468648.submit parsl.executors.status_handling INFO: Allocated block ID 1281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1281 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1281 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1281.1709793892.7473047.submit parsl.executors.status_handling INFO: Allocated block ID 1282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1282 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1282 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1282.1709793892.7477322.submit parsl.executors.status_handling INFO: Allocated block ID 1283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1283 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1283 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1283.1709793892.776462.submit parsl.executors.status_handling INFO: Allocated block ID 1284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1284 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1284 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1284.1709793892.7769268.submit parsl.executors.status_handling INFO: Allocated block ID 1285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1285 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1285 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1285.1709793892.777365.submit parsl.executors.status_handling INFO: Allocated block ID 1286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1286 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1286 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1286.1709793892.7777953.submit parsl.executors.status_handling INFO: Allocated block ID 1287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1287 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1287 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1287.1709793892.7782354.submit parsl.executors.status_handling INFO: Allocated block ID 1288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1288 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1288 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1288.1709793892.7786593.submit parsl.executors.status_handling INFO: Allocated block ID 1289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1289 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1289 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1289.1709793892.779092.submit parsl.executors.status_handling INFO: Allocated block ID 1290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1290 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1290 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1290.1709793892.7795146.submit parsl.executors.status_handling INFO: Allocated block ID 1291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1291 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1291 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1291.1709793892.7799425.submit parsl.executors.status_handling INFO: Allocated block ID 1292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1292 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1292 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1292.1709793892.7803676.submit parsl.executors.status_handling INFO: Allocated block ID 1293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1293 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1293 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1293.1709793892.7807982.submit parsl.executors.status_handling INFO: Allocated block ID 1294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1294 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1294 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1294.1709793892.7812307.submit parsl.executors.status_handling INFO: Allocated block ID 1295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1295 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1295 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1295.1709793892.7816575.submit parsl.executors.status_handling INFO: Allocated block ID 1296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1296 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1296 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1296.1709793892.7820837.submit parsl.executors.status_handling INFO: Allocated block ID 1297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1297 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1297 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1297.1709793892.782505.submit parsl.executors.status_handling INFO: Allocated block ID 1298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1298 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1298 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1298.1709793892.7829351.submit parsl.executors.status_handling INFO: Allocated block ID 1299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1299 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1299 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1299.1709793892.7833586.submit parsl.executors.status_handling INFO: Allocated block ID 1300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1300 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1300 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1300.1709793892.7837782.submit parsl.executors.status_handling INFO: Allocated block ID 1301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1301 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1301 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1301.1709793892.7842064.submit parsl.executors.status_handling INFO: Allocated block ID 1302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1302 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1302 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1302.1709793892.784623.submit parsl.executors.status_handling INFO: Allocated block ID 1303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1303 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1303 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1303.1709793892.7952194.submit parsl.executors.status_handling INFO: Allocated block ID 1304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1304 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1304 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1304.1709793892.7956731.submit parsl.executors.status_handling INFO: Allocated block ID 1305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1305 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1305 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1305.1709793892.796113.submit parsl.executors.status_handling INFO: Allocated block ID 1306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1306 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1306 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1306.1709793892.7965481.submit parsl.executors.status_handling INFO: Allocated block ID 1307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1307 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1307 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1307.1709793892.7969782.submit parsl.executors.status_handling INFO: Allocated block ID 1308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1308 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1308 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1308.1709793892.7973998.submit parsl.executors.status_handling INFO: Allocated block ID 1309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1309 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1309 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1309.1709793892.7978299.submit parsl.executors.status_handling INFO: Allocated block ID 1310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1310 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1310 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1310.1709793892.798259.submit parsl.executors.status_handling INFO: Allocated block ID 1311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1311 2024-03-06 22:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1311 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1311.1709793892.7986803.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cad518efa9945364955cf9de8442cf3 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1da186b6-58cd-4058-86de-f4bc7eab6a76_biasIsr_1123838_47.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123838/1da186b6-58cd-4058-86de-f4bc7eab6a76_biasIsr_1123838_47.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d2c8f57888aa229b386038cd22b867 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b31f9cdc-2be6-42ac-8924-cfa5654e0586_biasIsr_1123835_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/b31f9cdc-2be6-42ac-8924-cfa5654e0586_biasIsr_1123835_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1358 parsl.dataflow.strategy DEBUG: Executor multi has 1358 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 1312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1312 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1312 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1312.1709793897.6296651.submit parsl.executors.status_handling INFO: Allocated block ID 1313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1313 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1313 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1313.1709793897.631755.submit parsl.executors.status_handling INFO: Allocated block ID 1314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1314 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1314 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1314.1709793897.6337447.submit parsl.executors.status_handling INFO: Allocated block ID 1315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1315 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1315 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1315.1709793897.635697.submit parsl.executors.status_handling INFO: Allocated block ID 1316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1316 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1316 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1316.1709793897.6376545.submit parsl.executors.status_handling INFO: Allocated block ID 1317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1317 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1317 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1317.1709793897.6396234.submit parsl.executors.status_handling INFO: Allocated block ID 1318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1318 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1318 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1318.1709793897.6405706.submit parsl.executors.status_handling INFO: Allocated block ID 1319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1319 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1319 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1319.1709793897.6413891.submit parsl.executors.status_handling INFO: Allocated block ID 1320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1320 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1320 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1320.1709793897.642208.submit parsl.executors.status_handling INFO: Allocated block ID 1321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1321 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1321 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1321.1709793897.64302.submit parsl.executors.status_handling INFO: Allocated block ID 1322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1322 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1322 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1322.1709793897.64383.submit parsl.executors.status_handling INFO: Allocated block ID 1323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1323 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1323 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1323.1709793897.6446335.submit parsl.executors.status_handling INFO: Allocated block ID 1324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1324 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1324 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1324.1709793897.6454682.submit parsl.executors.status_handling INFO: Allocated block ID 1325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1325 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1325 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1325.1709793897.6463048.submit parsl.executors.status_handling INFO: Allocated block ID 1326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1326 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1326 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1326.1709793897.6471186.submit parsl.executors.status_handling INFO: Allocated block ID 1327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1327 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1327 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1327.1709793897.6479342.submit parsl.executors.status_handling INFO: Allocated block ID 1328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1328 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1328 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1328.1709793897.648741.submit parsl.executors.status_handling INFO: Allocated block ID 1329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1329 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1329 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1329.1709793897.6495528.submit parsl.executors.status_handling INFO: Allocated block ID 1330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1330 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1330 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1330.1709793897.6500514.submit parsl.executors.status_handling INFO: Allocated block ID 1331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1331 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1331 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1331.1709793897.6504683.submit parsl.executors.status_handling INFO: Allocated block ID 1332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1332 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1332 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1332.1709793897.6508946.submit parsl.executors.status_handling INFO: Allocated block ID 1333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1333 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1333 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1333.1709793897.6513116.submit parsl.executors.status_handling INFO: Allocated block ID 1334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1334 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1334 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1334.1709793897.6517284.submit parsl.executors.status_handling INFO: Allocated block ID 1335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1335 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1335 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1335.1709793897.652152.submit parsl.executors.status_handling INFO: Allocated block ID 1336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1336 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1336 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1336.1709793897.6525784.submit parsl.executors.status_handling INFO: Allocated block ID 1337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1337 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1337 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1337.1709793897.6530097.submit parsl.executors.status_handling INFO: Allocated block ID 1338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1338 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1338 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1338.1709793897.6534302.submit parsl.executors.status_handling INFO: Allocated block ID 1339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1339 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1339 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1339.1709793897.6538517.submit parsl.executors.status_handling INFO: Allocated block ID 1340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1340 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1340 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1340.1709793897.6542716.submit parsl.executors.status_handling INFO: Allocated block ID 1341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1341 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1341 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1341.1709793897.6546857.submit parsl.executors.status_handling INFO: Allocated block ID 1342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1342 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1342 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1342.1709793897.6551137.submit parsl.executors.status_handling INFO: Allocated block ID 1343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1343 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1343 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1343.1709793897.6555307.submit parsl.executors.status_handling INFO: Allocated block ID 1344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1344 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1344 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1344.1709793897.6559548.submit parsl.executors.status_handling INFO: Allocated block ID 1345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1345 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1345 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1345.1709793897.6563754.submit parsl.executors.status_handling INFO: Allocated block ID 1346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1346 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1346 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1346.1709793897.6567957.submit parsl.executors.status_handling INFO: Allocated block ID 1347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1347 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1347 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1347.1709793897.6572242.submit parsl.executors.status_handling INFO: Allocated block ID 1348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1348 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1348 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1348.1709793897.6576507.submit parsl.executors.status_handling INFO: Allocated block ID 1349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1349 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1349 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1349.1709793897.658075.submit parsl.executors.status_handling INFO: Allocated block ID 1350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1350 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1350 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1350.1709793897.6584942.submit parsl.executors.status_handling INFO: Allocated block ID 1351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1351 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1351 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1351.1709793897.65892.submit parsl.executors.status_handling INFO: Allocated block ID 1352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1352 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1352 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1352.1709793897.6593368.submit parsl.executors.status_handling INFO: Allocated block ID 1353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1353 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1353 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1353.1709793897.6597536.submit parsl.executors.status_handling INFO: Allocated block ID 1354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1354 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1354 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1354.1709793897.6601808.submit parsl.executors.status_handling INFO: Allocated block ID 1355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1355 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1355 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1355.1709793897.6605968.submit parsl.executors.status_handling INFO: Allocated block ID 1356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1356 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1356 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1356.1709793897.6610203.submit parsl.executors.status_handling INFO: Allocated block ID 1357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1357 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1357 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1357.1709793897.6614397.submit parsl.executors.status_handling INFO: Allocated block ID 1358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1358 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1358 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1358.1709793897.6618629.submit parsl.executors.status_handling INFO: Allocated block ID 1359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1359 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1359 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1359.1709793897.6622806.submit parsl.executors.status_handling INFO: Allocated block ID 1360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1360 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1360 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1360.1709793897.6627069.submit parsl.executors.status_handling INFO: Allocated block ID 1361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1361 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1361 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1361.1709793897.6631355.submit parsl.executors.status_handling INFO: Allocated block ID 1362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1362 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1362 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1362.1709793897.6635542.submit parsl.executors.status_handling INFO: Allocated block ID 1363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1363 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1363 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1363.1709793897.6640012.submit parsl.executors.status_handling INFO: Allocated block ID 1364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1364 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1364 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1364.1709793897.664445.submit parsl.executors.status_handling INFO: Allocated block ID 1365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1365 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1365 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1365.1709793897.667039.submit parsl.executors.status_handling INFO: Allocated block ID 1366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1366 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1366 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1366.1709793897.6675365.submit parsl.executors.status_handling INFO: Allocated block ID 1367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1367 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1367 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1367.1709793897.6680665.submit parsl.executors.status_handling INFO: Allocated block ID 1368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1368 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1368 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1368.1709793897.6685169.submit parsl.executors.status_handling INFO: Allocated block ID 1369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1369 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1369 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1369.1709793897.66895.submit parsl.executors.status_handling INFO: Allocated block ID 1370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1370 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1370 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1370.1709793897.669373.submit parsl.executors.status_handling INFO: Allocated block ID 1371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1371 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1371 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1371.1709793897.6697934.submit parsl.executors.status_handling INFO: Allocated block ID 1372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1372 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1372 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1372.1709793897.6702383.submit parsl.executors.status_handling INFO: Allocated block ID 1373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1373 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1373 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1373.1709793897.6706605.submit parsl.executors.status_handling INFO: Allocated block ID 1374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1374 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1374 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1374.1709793897.6710906.submit parsl.executors.status_handling INFO: Allocated block ID 1375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1375 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1375 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1375.1709793897.6715093.submit parsl.executors.status_handling INFO: Allocated block ID 1376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1376 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1376 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1376.1709793897.6719377.submit parsl.executors.status_handling INFO: Allocated block ID 1377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1377 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1377 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1377.1709793897.6723585.submit parsl.executors.status_handling INFO: Allocated block ID 1378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1378 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1378 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1378.1709793897.6728005.submit parsl.executors.status_handling INFO: Allocated block ID 1379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1379 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1379 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1379.1709793897.6732347.submit parsl.executors.status_handling INFO: Allocated block ID 1380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1380 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1380 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1380.1709793897.6736617.submit parsl.executors.status_handling INFO: Allocated block ID 1381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1381 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1381 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1381.1709793897.6740935.submit parsl.executors.status_handling INFO: Allocated block ID 1382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1382 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1382 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1382.1709793897.6745121.submit parsl.executors.status_handling INFO: Allocated block ID 1383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1383 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1383 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1383.1709793897.6749346.submit parsl.executors.status_handling INFO: Allocated block ID 1384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1384 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1384 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1384.1709793897.6753604.submit parsl.executors.status_handling INFO: Allocated block ID 1385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1385 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1385 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1385.1709793897.6757784.submit parsl.executors.status_handling INFO: Allocated block ID 1386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1386 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1386 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1386.1709793897.6762056.submit parsl.executors.status_handling INFO: Allocated block ID 1387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1387 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1387 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1387.1709793897.6766183.submit parsl.executors.status_handling INFO: Allocated block ID 1388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1388 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1388 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1388.1709793897.677047.submit parsl.executors.status_handling INFO: Allocated block ID 1389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1389 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1389 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1389.1709793897.6774628.submit parsl.executors.status_handling INFO: Allocated block ID 1390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1390 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1390 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1390.1709793897.6778882.submit parsl.executors.status_handling INFO: Allocated block ID 1391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1391 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1391 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1391.1709793897.6783042.submit parsl.executors.status_handling INFO: Allocated block ID 1392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1392 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1392 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1392.1709793897.6787276.submit parsl.executors.status_handling INFO: Allocated block ID 1393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1393 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1393 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1393.1709793897.6791544.submit parsl.executors.status_handling INFO: Allocated block ID 1394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1394 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1394 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1394.1709793897.679586.submit parsl.executors.status_handling INFO: Allocated block ID 1395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1395 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1395 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1395.1709793897.680077.submit parsl.executors.status_handling INFO: Allocated block ID 1396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1396 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1396 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1396.1709793897.680523.submit parsl.executors.status_handling INFO: Allocated block ID 1397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1397 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1397 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1397.1709793897.6809726.submit parsl.executors.status_handling INFO: Allocated block ID 1398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1398 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1398 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1398.1709793897.6813943.submit parsl.executors.status_handling INFO: Allocated block ID 1399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1399 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1399 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1399.1709793897.681819.submit parsl.executors.status_handling INFO: Allocated block ID 1400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1400 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1400 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1400.1709793897.682247.submit parsl.executors.status_handling INFO: Allocated block ID 1401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1401 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1401 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1401.1709793897.682661.submit parsl.executors.status_handling INFO: Allocated block ID 1402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1402 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1402 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1402.1709793897.683093.submit parsl.executors.status_handling INFO: Allocated block ID 1403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1403 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1403 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1403.1709793897.6835155.submit parsl.executors.status_handling INFO: Allocated block ID 1404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1404 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1404 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1404.1709793897.6839442.submit parsl.executors.status_handling INFO: Allocated block ID 1405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1405 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1405 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1405.1709793897.6843889.submit parsl.executors.status_handling INFO: Allocated block ID 1406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1406 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1406 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1406.1709793897.6848223.submit parsl.executors.status_handling INFO: Allocated block ID 1407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1407 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1407 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1407.1709793897.6852493.submit parsl.executors.status_handling INFO: Allocated block ID 1408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1408 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1408 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1408.1709793897.6856763.submit parsl.executors.status_handling INFO: Allocated block ID 1409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1409 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1409 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1409.1709793897.6861138.submit parsl.executors.status_handling INFO: Allocated block ID 1410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1410 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1410 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1410.1709793897.6865406.submit parsl.executors.status_handling INFO: Allocated block ID 1411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1411 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1411 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1411.1709793897.6869698.submit parsl.executors.status_handling INFO: Allocated block ID 1412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1412 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1412 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1412.1709793897.6873965.submit parsl.executors.status_handling INFO: Allocated block ID 1413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1413 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1413 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1413.1709793897.6878235.submit parsl.executors.status_handling INFO: Allocated block ID 1414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1414 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1414 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1414.1709793897.6882498.submit parsl.executors.status_handling INFO: Allocated block ID 1415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1415 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1415 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1415.1709793897.6886673.submit parsl.executors.status_handling INFO: Allocated block ID 1416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1416 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1416 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1416.1709793897.68909.submit parsl.executors.status_handling INFO: Allocated block ID 1417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1417 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1417 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1417.1709793897.6895103.submit parsl.executors.status_handling INFO: Allocated block ID 1418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1418 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1418 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1418.1709793897.6899352.submit parsl.executors.status_handling INFO: Allocated block ID 1419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1419 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1419 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1419.1709793897.690351.submit parsl.executors.status_handling INFO: Allocated block ID 1420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1420 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1420 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1420.1709793897.6907794.submit parsl.executors.status_handling INFO: Allocated block ID 1421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1421 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1421 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1421.1709793897.6912048.submit parsl.executors.status_handling INFO: Allocated block ID 1422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1422 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1422 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1422.1709793897.6916265.submit parsl.executors.status_handling INFO: Allocated block ID 1423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1423 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1423 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1423.1709793897.6920602.submit parsl.executors.status_handling INFO: Allocated block ID 1424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1424 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1424 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1424.1709793897.6925807.submit parsl.executors.status_handling INFO: Allocated block ID 1425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1425 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1425 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1425.1709793897.6930122.submit parsl.executors.status_handling INFO: Allocated block ID 1426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1426 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1426 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1426.1709793897.6934319.submit parsl.executors.status_handling INFO: Allocated block ID 1427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1427 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1427 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1427.1709793897.6938562.submit parsl.executors.status_handling INFO: Allocated block ID 1428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1428 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1428 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1428.1709793897.6942725.submit parsl.executors.status_handling INFO: Allocated block ID 1429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1429 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1429 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1429.1709793897.6946921.submit parsl.executors.status_handling INFO: Allocated block ID 1430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1430 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1430 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1430.1709793897.6951182.submit parsl.executors.status_handling INFO: Allocated block ID 1431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1431 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1431 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1431.1709793897.695544.submit parsl.executors.status_handling INFO: Allocated block ID 1432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1432 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1432 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1432.1709793897.695982.submit parsl.executors.status_handling INFO: Allocated block ID 1433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1433 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1433 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1433.1709793897.69641.submit parsl.executors.status_handling INFO: Allocated block ID 1434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1434 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1434 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1434.1709793897.696842.submit parsl.executors.status_handling INFO: Allocated block ID 1435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1435 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1435 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1435.1709793897.6972613.submit parsl.executors.status_handling INFO: Allocated block ID 1436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1436 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1436 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1436.1709793897.6976614.submit parsl.executors.status_handling INFO: Allocated block ID 1437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1437 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1437 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1437.1709793897.69807.submit parsl.executors.status_handling INFO: Allocated block ID 1438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1438 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1438 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1438.1709793897.6985009.submit parsl.executors.status_handling INFO: Allocated block ID 1439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1439 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1439 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1439.1709793897.6989398.submit parsl.executors.status_handling INFO: Allocated block ID 1440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1440 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1440 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1440.1709793897.6993642.submit parsl.executors.status_handling INFO: Allocated block ID 1441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1441 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1441 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1441.1709793897.6997905.submit parsl.executors.status_handling INFO: Allocated block ID 1442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1442 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1442 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1442.1709793897.700222.submit parsl.executors.status_handling INFO: Allocated block ID 1443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1443 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1443 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1443.1709793897.700651.submit parsl.executors.status_handling INFO: Allocated block ID 1444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1444 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1444 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1444.1709793897.7010918.submit parsl.executors.status_handling INFO: Allocated block ID 1445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1445 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1445 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1445.1709793897.7015183.submit parsl.executors.status_handling INFO: Allocated block ID 1446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1446 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1446 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1446.1709793897.7019503.submit parsl.executors.status_handling INFO: Allocated block ID 1447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1447 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1447 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1447.1709793897.7023695.submit parsl.executors.status_handling INFO: Allocated block ID 1448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1448 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1448 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1448.1709793897.702794.submit parsl.executors.status_handling INFO: Allocated block ID 1449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1449 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1449 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1449.1709793897.7032273.submit parsl.executors.status_handling INFO: Allocated block ID 1450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1450 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1450 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1450.1709793897.7036517.submit parsl.executors.status_handling INFO: Allocated block ID 1451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1451 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1451 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1451.1709793897.7040825.submit parsl.executors.status_handling INFO: Allocated block ID 1452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1452 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1452 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1452.1709793897.7045257.submit parsl.executors.status_handling INFO: Allocated block ID 1453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1453 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1453 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1453.1709793897.7049582.submit parsl.executors.status_handling INFO: Allocated block ID 1454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1454 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1454 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1454.1709793897.705386.submit parsl.executors.status_handling INFO: Allocated block ID 1455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1455 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1455 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1455.1709793897.7058153.submit parsl.executors.status_handling INFO: Allocated block ID 1456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1456 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1456 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1456.1709793897.7062516.submit parsl.executors.status_handling INFO: Allocated block ID 1457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1457 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1457 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1457.1709793897.7066824.submit parsl.executors.status_handling INFO: Allocated block ID 1458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1458 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1458 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1458.1709793897.7071161.submit parsl.executors.status_handling INFO: Allocated block ID 1459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1459 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1459 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1459.1709793897.7075408.submit parsl.executors.status_handling INFO: Allocated block ID 1460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1460 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1460 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1460.1709793897.7079692.submit parsl.executors.status_handling INFO: Allocated block ID 1461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1461 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1461 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1461.1709793897.7083924.submit parsl.executors.status_handling INFO: Allocated block ID 1462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1462 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1462 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1462.1709793897.7088244.submit parsl.executors.status_handling INFO: Allocated block ID 1463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1463 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1463 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1463.1709793897.7092483.submit parsl.executors.status_handling INFO: Allocated block ID 1464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1464 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1464 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1464.1709793897.7096705.submit parsl.executors.status_handling INFO: Allocated block ID 1465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1465 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1465 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1465.1709793897.7100954.submit parsl.executors.status_handling INFO: Allocated block ID 1466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1466 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1466 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1466.1709793897.7105207.submit parsl.executors.status_handling INFO: Allocated block ID 1467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1467 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1467 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1467.1709793897.7109606.submit parsl.executors.status_handling INFO: Allocated block ID 1468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1468 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1468 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1468.1709793897.7113945.submit parsl.executors.status_handling INFO: Allocated block ID 1469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1469 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1469 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1469.1709793897.7118304.submit parsl.executors.status_handling INFO: Allocated block ID 1470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1470 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1470 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1470.1709793897.7122822.submit parsl.executors.status_handling INFO: Allocated block ID 1471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1471 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1471 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1471.1709793897.7127163.submit parsl.executors.status_handling INFO: Allocated block ID 1472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1472 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1472 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1472.1709793897.7131517.submit parsl.executors.status_handling INFO: Allocated block ID 1473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1473 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1473 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1473.1709793897.7135942.submit parsl.executors.status_handling INFO: Allocated block ID 1474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1474 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1474 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1474.1709793897.7140245.submit parsl.executors.status_handling INFO: Allocated block ID 1475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1475 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1475 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1475.1709793897.7144446.submit parsl.executors.status_handling INFO: Allocated block ID 1476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1476 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1476 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1476.1709793897.7148705.submit parsl.executors.status_handling INFO: Allocated block ID 1477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1477 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1477 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1477.1709793897.715291.submit parsl.executors.status_handling INFO: Allocated block ID 1478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1478 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1478 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1478.1709793897.7157137.submit parsl.executors.status_handling INFO: Allocated block ID 1479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1479 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1479 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1479.1709793897.7161415.submit parsl.executors.status_handling INFO: Allocated block ID 1480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1480 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1480 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1480.1709793897.7165713.submit parsl.executors.status_handling INFO: Allocated block ID 1481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1481 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1481 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1481.1709793897.7170014.submit parsl.executors.status_handling INFO: Allocated block ID 1482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1482 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1482 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1482.1709793897.717426.submit parsl.executors.status_handling INFO: Allocated block ID 1483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1483 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1483 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1483.1709793897.7178555.submit parsl.executors.status_handling INFO: Allocated block ID 1484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1484 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1484 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1484.1709793897.7182784.submit parsl.executors.status_handling INFO: Allocated block ID 1485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1485 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1485 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1485.1709793897.7186975.submit parsl.executors.status_handling INFO: Allocated block ID 1486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1486 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1486 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1486.1709793897.7191277.submit parsl.executors.status_handling INFO: Allocated block ID 1487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1487 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1487 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1487.1709793897.7195466.submit parsl.executors.status_handling INFO: Allocated block ID 1488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1488 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1488 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1488.1709793897.7199762.submit parsl.executors.status_handling INFO: Allocated block ID 1489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1489 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1489 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1489.1709793897.7203958.submit parsl.executors.status_handling INFO: Allocated block ID 1490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1490 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1490 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1490.1709793897.720819.submit parsl.executors.status_handling INFO: Allocated block ID 1491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1491 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1491 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1491.1709793897.721236.submit parsl.executors.status_handling INFO: Allocated block ID 1492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1492 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1492 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1492.1709793897.7216623.submit parsl.executors.status_handling INFO: Allocated block ID 1493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1493 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1493 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1493.1709793897.7220888.submit parsl.executors.status_handling INFO: Allocated block ID 1494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1494 2024-03-06 22:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1494 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1494.1709793897.7225094.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1358 parsl.dataflow.strategy DEBUG: Executor multi has 1358 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 1495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1495 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1495 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1495.1709793902.6299794.submit parsl.executors.status_handling INFO: Allocated block ID 1496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1496 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1496 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1496.1709793902.6304803.submit parsl.executors.status_handling INFO: Allocated block ID 1497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1497 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1497 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1497.1709793902.6309314.submit parsl.executors.status_handling INFO: Allocated block ID 1498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1498 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1498 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1498.1709793902.6313605.submit parsl.executors.status_handling INFO: Allocated block ID 1499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1499 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1499 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1499.1709793902.6317847.submit parsl.executors.status_handling INFO: Allocated block ID 1500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1500 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1500 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1500.1709793902.6322145.submit parsl.executors.status_handling INFO: Allocated block ID 1501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1501 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1501 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1501.1709793902.63264.submit parsl.executors.status_handling INFO: Allocated block ID 1502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1502 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1502 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1502.1709793902.6330793.submit parsl.executors.status_handling INFO: Allocated block ID 1503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1503 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1503 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1503.1709793902.6335042.submit parsl.executors.status_handling INFO: Allocated block ID 1504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1504 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1504 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1504.1709793902.6339352.submit parsl.executors.status_handling INFO: Allocated block ID 1505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1505 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1505 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1505.1709793902.6343598.submit parsl.executors.status_handling INFO: Allocated block ID 1506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1506 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1506 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1506.1709793902.6347826.submit parsl.executors.status_handling INFO: Allocated block ID 1507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1507 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1507 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1507.1709793902.6352155.submit parsl.executors.status_handling INFO: Allocated block ID 1508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1508 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1508 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1508.1709793902.6356468.submit parsl.executors.status_handling INFO: Allocated block ID 1509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1509 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1509 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1509.1709793902.636084.submit parsl.executors.status_handling INFO: Allocated block ID 1510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1510 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1510 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1510.1709793902.6365075.submit parsl.executors.status_handling INFO: Allocated block ID 1511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1511 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1511 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1511.1709793902.6369388.submit parsl.executors.status_handling INFO: Allocated block ID 1512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1512 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1512 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1512.1709793902.6373687.submit parsl.executors.status_handling INFO: Allocated block ID 1513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1513 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1513 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1513.1709793902.637791.submit parsl.executors.status_handling INFO: Allocated block ID 1514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1514 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1514 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1514.1709793902.638235.submit parsl.executors.status_handling INFO: Allocated block ID 1515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1515 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1515 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1515.1709793902.6386557.submit parsl.executors.status_handling INFO: Allocated block ID 1516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1516 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1516 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1516.1709793902.6390889.submit parsl.executors.status_handling INFO: Allocated block ID 1517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1517 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1517 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1517.1709793902.639511.submit parsl.executors.status_handling INFO: Allocated block ID 1518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1518 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1518 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1518.1709793902.6399403.submit parsl.executors.status_handling INFO: Allocated block ID 1519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1519 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1519 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1519.1709793902.6403646.submit parsl.executors.status_handling INFO: Allocated block ID 1520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1520 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1520 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1520.1709793902.6407876.submit parsl.executors.status_handling INFO: Allocated block ID 1521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1521 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1521 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1521.1709793902.6412175.submit parsl.executors.status_handling INFO: Allocated block ID 1522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1522 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1522 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1522.1709793902.641638.submit parsl.executors.status_handling INFO: Allocated block ID 1523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1523 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1523 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1523.1709793902.6552558.submit parsl.executors.status_handling INFO: Allocated block ID 1524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1524 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1524 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1524.1709793902.6557443.submit parsl.executors.status_handling INFO: Allocated block ID 1525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1525 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1525 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1525.1709793902.6561868.submit parsl.executors.status_handling INFO: Allocated block ID 1526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1526 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1526 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1526.1709793902.6566186.submit parsl.executors.status_handling INFO: Allocated block ID 1527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1527 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1527 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1527.1709793902.6570551.submit parsl.executors.status_handling INFO: Allocated block ID 1528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1528 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1528 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1528.1709793902.6574767.submit parsl.executors.status_handling INFO: Allocated block ID 1529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1529 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1529 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1529.1709793902.657904.submit parsl.executors.status_handling INFO: Allocated block ID 1530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1530 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1530 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1530.1709793902.658324.submit parsl.executors.status_handling INFO: Allocated block ID 1531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1531 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1531 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1531.1709793902.658741.submit parsl.executors.status_handling INFO: Allocated block ID 1532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1532 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1532 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1532.1709793902.659171.submit parsl.executors.status_handling INFO: Allocated block ID 1533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1533 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1533 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1533.1709793902.6595905.submit parsl.executors.status_handling INFO: Allocated block ID 1534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1534 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1534 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1534.1709793902.6600194.submit parsl.executors.status_handling INFO: Allocated block ID 1535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1535 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1535 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1535.1709793902.6604414.submit parsl.executors.status_handling INFO: Allocated block ID 1536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1536 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1536 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1536.1709793902.6608667.submit parsl.executors.status_handling INFO: Allocated block ID 1537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1537 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1537 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1537.1709793902.661294.submit parsl.executors.status_handling INFO: Allocated block ID 1538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1538 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1538 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1538.1709793902.6617243.submit parsl.executors.status_handling INFO: Allocated block ID 1539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1539 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1539 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1539.1709793902.6621583.submit parsl.executors.status_handling INFO: Allocated block ID 1540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1540 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1540 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1540.1709793902.6625822.submit parsl.executors.status_handling INFO: Allocated block ID 1541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1541 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1541 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1541.1709793902.6630108.submit parsl.executors.status_handling INFO: Allocated block ID 1542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1542 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1542 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1542.1709793902.6634326.submit parsl.executors.status_handling INFO: Allocated block ID 1543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1543 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1543 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1543.1709793902.6638682.submit parsl.executors.status_handling INFO: Allocated block ID 1544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1544 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1544 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1544.1709793902.6642945.submit parsl.executors.status_handling INFO: Allocated block ID 1545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1545 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1545 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1545.1709793902.6647089.submit parsl.executors.status_handling INFO: Allocated block ID 1546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1546 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1546 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1546.1709793902.665137.submit parsl.executors.status_handling INFO: Allocated block ID 1547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1547 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1547 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1547.1709793902.6655567.submit parsl.executors.status_handling INFO: Allocated block ID 1548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1548 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1548 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1548.1709793902.6659791.submit parsl.executors.status_handling INFO: Allocated block ID 1549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1549 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1549 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1549.1709793902.6663973.submit parsl.executors.status_handling INFO: Allocated block ID 1550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1550 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1550 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1550.1709793902.666829.submit parsl.executors.status_handling INFO: Allocated block ID 1551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1551 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1551 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1551.1709793902.667249.submit parsl.executors.status_handling INFO: Allocated block ID 1552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1552 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1552 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1552.1709793902.6676676.submit parsl.executors.status_handling INFO: Allocated block ID 1553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1553 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1553 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1553.1709793902.6812344.submit parsl.executors.status_handling INFO: Allocated block ID 1554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1554 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1554 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1554.1709793902.6816952.submit parsl.executors.status_handling INFO: Allocated block ID 1555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1555 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1555 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1555.1709793902.682128.submit parsl.executors.status_handling INFO: Allocated block ID 1556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1556 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1556 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1556.1709793902.682555.submit parsl.executors.status_handling INFO: Allocated block ID 1557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1557 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1557 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1557.1709793902.6829867.submit parsl.executors.status_handling INFO: Allocated block ID 1558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1558 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1558 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1558.1709793902.6834135.submit parsl.executors.status_handling INFO: Allocated block ID 1559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1559 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1559 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1559.1709793902.6838448.submit parsl.executors.status_handling INFO: Allocated block ID 1560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1560 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1560 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1560.1709793902.684266.submit parsl.executors.status_handling INFO: Allocated block ID 1561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1561 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1561 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1561.1709793902.6846833.submit parsl.executors.status_handling INFO: Allocated block ID 1562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1562 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1562 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1562.1709793902.6851182.submit parsl.executors.status_handling INFO: Allocated block ID 1563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1563 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1563 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1563.1709793902.6855671.submit parsl.executors.status_handling INFO: Allocated block ID 1564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1564 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1564 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1564.1709793902.686009.submit parsl.executors.status_handling INFO: Allocated block ID 1565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1565 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1565 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1565.1709793902.6864412.submit parsl.executors.status_handling INFO: Allocated block ID 1566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1566 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1566 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1566.1709793902.686864.submit parsl.executors.status_handling INFO: Allocated block ID 1567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1567 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1567 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1567.1709793902.6872833.submit parsl.executors.status_handling INFO: Allocated block ID 1568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1568 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1568 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1568.1709793902.6877077.submit parsl.executors.status_handling INFO: Allocated block ID 1569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1569 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1569 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1569.1709793902.6881328.submit parsl.executors.status_handling INFO: Allocated block ID 1570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1570 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1570 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1570.1709793902.688551.submit parsl.executors.status_handling INFO: Allocated block ID 1571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1571 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1571 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1571.1709793902.688977.submit parsl.executors.status_handling INFO: Allocated block ID 1572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1572 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1572 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1572.1709793902.6894028.submit parsl.executors.status_handling INFO: Allocated block ID 1573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1573 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1573 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1573.1709793902.6898243.submit parsl.executors.status_handling INFO: Allocated block ID 1574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1574 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1574 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1574.1709793902.6902506.submit parsl.executors.status_handling INFO: Allocated block ID 1575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1575 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1575 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1575.1709793902.6906722.submit parsl.executors.status_handling INFO: Allocated block ID 1576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1576 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1576 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1576.1709793902.6910954.submit parsl.executors.status_handling INFO: Allocated block ID 1577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1577 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1577 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1577.1709793902.6915152.submit parsl.executors.status_handling INFO: Allocated block ID 1578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1578 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1578 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1578.1709793902.69194.submit parsl.executors.status_handling INFO: Allocated block ID 1579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1579 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1579 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1579.1709793902.692359.submit parsl.executors.status_handling INFO: Allocated block ID 1580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1580 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1580 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1580.1709793902.6927874.submit parsl.executors.status_handling INFO: Allocated block ID 1581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1581 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1581 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1581.1709793902.6932142.submit parsl.executors.status_handling INFO: Allocated block ID 1582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1582 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1582 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1582.1709793902.693631.submit parsl.executors.status_handling INFO: Allocated block ID 1583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1583 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1583 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1583.1709793902.707195.submit parsl.executors.status_handling INFO: Allocated block ID 1584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1584 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1584 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1584.1709793902.7076633.submit parsl.executors.status_handling INFO: Allocated block ID 1585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1585 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1585 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1585.1709793902.7080965.submit parsl.executors.status_handling INFO: Allocated block ID 1586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1586 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1586 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1586.1709793902.708528.submit parsl.executors.status_handling INFO: Allocated block ID 1587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1587 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1587 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1587.1709793902.7089689.submit parsl.executors.status_handling INFO: Allocated block ID 1588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1588 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1588 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1588.1709793902.7093961.submit parsl.executors.status_handling INFO: Allocated block ID 1589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1589 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1589 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1589.1709793902.7098203.submit parsl.executors.status_handling INFO: Allocated block ID 1590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1590 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1590 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1590.1709793902.7102423.submit parsl.executors.status_handling INFO: Allocated block ID 1591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1591 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1591 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1591.1709793902.7106576.submit parsl.executors.status_handling INFO: Allocated block ID 1592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1592 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1592 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1592.1709793902.7110884.submit parsl.executors.status_handling INFO: Allocated block ID 1593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1593 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1593 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1593.1709793902.7115078.submit parsl.executors.status_handling INFO: Allocated block ID 1594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1594 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1594 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1594.1709793902.7119288.submit parsl.executors.status_handling INFO: Allocated block ID 1595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1595 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1595 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1595.1709793902.712346.submit parsl.executors.status_handling INFO: Allocated block ID 1596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1596 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1596 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1596.1709793902.7127607.submit parsl.executors.status_handling INFO: Allocated block ID 1597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1597 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1597 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1597.1709793902.713186.submit parsl.executors.status_handling INFO: Allocated block ID 1598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1598 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1598 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1598.1709793902.7136142.submit parsl.executors.status_handling INFO: Allocated block ID 1599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1599 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1599 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1599.1709793902.714043.submit parsl.executors.status_handling INFO: Allocated block ID 1600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1600 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1600 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1600.1709793902.7144668.submit parsl.executors.status_handling INFO: Allocated block ID 1601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1601 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1601 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1601.1709793902.7148945.submit parsl.executors.status_handling INFO: Allocated block ID 1602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1602 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1602 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1602.1709793902.7153122.submit parsl.executors.status_handling INFO: Allocated block ID 1603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1603 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1603 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1603.1709793902.7157328.submit parsl.executors.status_handling INFO: Allocated block ID 1604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1604 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1604 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1604.1709793902.716162.submit parsl.executors.status_handling INFO: Allocated block ID 1605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1605 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1605 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1605.1709793902.7165804.submit parsl.executors.status_handling INFO: Allocated block ID 1606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1606 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1606 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1606.1709793902.717003.submit parsl.executors.status_handling INFO: Allocated block ID 1607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1607 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1607 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1607.1709793902.7174165.submit parsl.executors.status_handling INFO: Allocated block ID 1608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1608 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1608 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1608.1709793902.71784.submit parsl.executors.status_handling INFO: Allocated block ID 1609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1609 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1609 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1609.1709793902.718259.submit parsl.executors.status_handling INFO: Allocated block ID 1610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1610 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1610 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1610.1709793902.7186856.submit parsl.executors.status_handling INFO: Allocated block ID 1611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1611 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1611 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1611.1709793902.719113.submit parsl.executors.status_handling INFO: Allocated block ID 1612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1612 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1612 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1612.1709793902.719532.submit parsl.executors.status_handling INFO: Allocated block ID 1613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1613 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1613 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1613.1709793902.7330766.submit parsl.executors.status_handling INFO: Allocated block ID 1614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1614 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1614 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1614.1709793902.733579.submit parsl.executors.status_handling INFO: Allocated block ID 1615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1615 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1615 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1615.1709793902.734019.submit parsl.executors.status_handling INFO: Allocated block ID 1616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1616 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1616 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1616.1709793902.7344456.submit parsl.executors.status_handling INFO: Allocated block ID 1617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1617 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1617 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1617.1709793902.7348783.submit parsl.executors.status_handling INFO: Allocated block ID 1618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1618 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1618 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1618.1709793902.7353.submit parsl.executors.status_handling INFO: Allocated block ID 1619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1619 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1619 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1619.1709793902.7357183.submit parsl.executors.status_handling INFO: Allocated block ID 1620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1620 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1620 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1620.1709793902.7361443.submit parsl.executors.status_handling INFO: Allocated block ID 1621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1621 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1621 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1621.1709793902.736564.submit parsl.executors.status_handling INFO: Allocated block ID 1622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1622 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1622 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1622.1709793902.7370098.submit parsl.executors.status_handling INFO: Allocated block ID 1623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1623 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1623 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1623.1709793902.7374372.submit parsl.executors.status_handling INFO: Allocated block ID 1624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1624 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1624 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1624.1709793902.737868.submit parsl.executors.status_handling INFO: Allocated block ID 1625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1625 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1625 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1625.1709793902.7382932.submit parsl.executors.status_handling INFO: Allocated block ID 1626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1626 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1626 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1626.1709793902.7387125.submit parsl.executors.status_handling INFO: Allocated block ID 1627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1627 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1627 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1627.1709793902.7391417.submit parsl.executors.status_handling INFO: Allocated block ID 1628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1628 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1628 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1628.1709793902.7395627.submit parsl.executors.status_handling INFO: Allocated block ID 1629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1629 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1629 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1629.1709793902.7399855.submit parsl.executors.status_handling INFO: Allocated block ID 1630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1630 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1630 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1630.1709793902.7404058.submit parsl.executors.status_handling INFO: Allocated block ID 1631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1631 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1631 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1631.1709793902.740829.submit parsl.executors.status_handling INFO: Allocated block ID 1632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1632 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1632 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1632.1709793902.7412543.submit parsl.executors.status_handling INFO: Allocated block ID 1633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1633 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1633 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1633.1709793902.7416725.submit parsl.executors.status_handling INFO: Allocated block ID 1634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1634 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1634 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1634.1709793902.7421093.submit parsl.executors.status_handling INFO: Allocated block ID 1635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1635 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1635 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1635.1709793902.7425306.submit parsl.executors.status_handling INFO: Allocated block ID 1636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1636 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1636 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1636.1709793902.7429574.submit parsl.executors.status_handling INFO: Allocated block ID 1637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1637 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1637 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1637.1709793902.7433746.submit parsl.executors.status_handling INFO: Allocated block ID 1638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1638 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1638 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1638.1709793902.743793.submit parsl.executors.status_handling INFO: Allocated block ID 1639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1639 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1639 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1639.1709793902.744221.submit parsl.executors.status_handling INFO: Allocated block ID 1640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1640 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1640 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1640.1709793902.7446423.submit parsl.executors.status_handling INFO: Allocated block ID 1641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1641 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1641 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1641.1709793902.7450657.submit parsl.executors.status_handling INFO: Allocated block ID 1642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1642 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1642 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1642.1709793902.7454853.submit parsl.executors.status_handling INFO: Allocated block ID 1643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1643 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1643 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1643.1709793902.759031.submit parsl.executors.status_handling INFO: Allocated block ID 1644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1644 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1644 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1644.1709793902.759536.submit parsl.executors.status_handling INFO: Allocated block ID 1645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1645 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1645 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1645.1709793902.759984.submit parsl.executors.status_handling INFO: Allocated block ID 1646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1646 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1646 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1646.1709793902.7604198.submit parsl.executors.status_handling INFO: Allocated block ID 1647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1647 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1647 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1647.1709793902.7608566.submit parsl.executors.status_handling INFO: Allocated block ID 1648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1648 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1648 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1648.1709793902.7612824.submit parsl.executors.status_handling INFO: Allocated block ID 1649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1649 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1649 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1649.1709793902.7616994.submit parsl.executors.status_handling INFO: Allocated block ID 1650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1650 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1650 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1650.1709793902.762127.submit parsl.executors.status_handling INFO: Allocated block ID 1651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1651 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1651 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1651.1709793902.762545.submit parsl.executors.status_handling INFO: Allocated block ID 1652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1652 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1652 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1652.1709793902.7629719.submit parsl.executors.status_handling INFO: Allocated block ID 1653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1653 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1653 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1653.1709793902.7633946.submit parsl.executors.status_handling INFO: Allocated block ID 1654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1654 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1654 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1654.1709793902.7638216.submit parsl.executors.status_handling INFO: Allocated block ID 1655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1655 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1655 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1655.1709793902.7642455.submit parsl.executors.status_handling INFO: Allocated block ID 1656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1656 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1656 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1656.1709793902.7646651.submit parsl.executors.status_handling INFO: Allocated block ID 1657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1657 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1657 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1657.1709793902.7650957.submit parsl.executors.status_handling INFO: Allocated block ID 1658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1658 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1658 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1658.1709793902.7655277.submit parsl.executors.status_handling INFO: Allocated block ID 1659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1659 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1659 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1659.1709793902.7659574.submit parsl.executors.status_handling INFO: Allocated block ID 1660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1660 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1660 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1660.1709793902.7663796.submit parsl.executors.status_handling INFO: Allocated block ID 1661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1661 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1661 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1661.1709793902.7668002.submit parsl.executors.status_handling INFO: Allocated block ID 1662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1662 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1662 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1662.1709793902.7672265.submit parsl.executors.status_handling INFO: Allocated block ID 1663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1663 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1663 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1663.1709793902.767648.submit parsl.executors.status_handling INFO: Allocated block ID 1664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1664 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1664 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1664.1709793902.7680779.submit parsl.executors.status_handling INFO: Allocated block ID 1665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1665 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1665 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1665.1709793902.7684999.submit parsl.executors.status_handling INFO: Allocated block ID 1666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1666 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1666 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1666.1709793902.7689283.submit parsl.executors.status_handling INFO: Allocated block ID 1667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1667 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1667 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1667.1709793902.7693477.submit parsl.executors.status_handling INFO: Allocated block ID 1668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1668 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1668 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1668.1709793902.7697651.submit parsl.executors.status_handling INFO: Allocated block ID 1669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1669 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1669 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1669.1709793902.7701917.submit parsl.executors.status_handling INFO: Allocated block ID 1670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1670 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1670 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1670.1709793902.7706258.submit parsl.executors.status_handling INFO: Allocated block ID 1671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1671 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1671 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1671.1709793902.7710552.submit parsl.executors.status_handling INFO: Allocated block ID 1672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1672 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1672 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1672.1709793902.7714796.submit parsl.executors.status_handling INFO: Allocated block ID 1673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1673 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1673 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1673.1709793902.7850041.submit parsl.executors.status_handling INFO: Allocated block ID 1674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1674 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1674 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1674.1709793902.7855046.submit parsl.executors.status_handling INFO: Allocated block ID 1675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1675 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1675 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1675.1709793902.7859406.submit parsl.executors.status_handling INFO: Allocated block ID 1676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1676 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1676 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1676.1709793902.7863693.submit parsl.executors.status_handling INFO: Allocated block ID 1677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1677 2024-03-06 22:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1677 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1677.1709793902.786798.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1e0a9fd3ec087d4e4bcdb7d1ca823d with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0f6a620d-a83c-4b68-8744-a9f5b0c01f5c_biasIsr_1123731_47.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/0f6a620d-a83c-4b68-8744-a9f5b0c01f5c_biasIsr_1123731_47.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4e0dcb56351862d2cff0977b4523017 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1cad6b7f-0fb2-4953-97b8-8c86ed50b6eb_biasIsr_1123729_47.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123729/1cad6b7f-0fb2-4953-97b8-8c86ed50b6eb_biasIsr_1123729_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1356 parsl.dataflow.strategy DEBUG: Executor multi has 1356 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 1678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1678 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1678 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1678.1709793907.6616013.submit parsl.executors.status_handling INFO: Allocated block ID 1679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1679 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1679 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1679.1709793907.6621263.submit parsl.executors.status_handling INFO: Allocated block ID 1680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1680 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1680 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1680.1709793907.6625783.submit parsl.executors.status_handling INFO: Allocated block ID 1681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1681 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1681 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1681.1709793907.6630285.submit parsl.executors.status_handling INFO: Allocated block ID 1682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1682 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1682 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1682.1709793907.663465.submit parsl.executors.status_handling INFO: Allocated block ID 1683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1683 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1683 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1683.1709793907.663907.submit parsl.executors.status_handling INFO: Allocated block ID 1684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1684 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1684 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1684.1709793907.6643457.submit parsl.executors.status_handling INFO: Allocated block ID 1685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1685 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1685 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1685.1709793907.664771.submit parsl.executors.status_handling INFO: Allocated block ID 1686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1686 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1686 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1686.1709793907.6652105.submit parsl.executors.status_handling INFO: Allocated block ID 1687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1687 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1687 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1687.1709793907.6656353.submit parsl.executors.status_handling INFO: Allocated block ID 1688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1688 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1688 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1688.1709793907.666066.submit parsl.executors.status_handling INFO: Allocated block ID 1689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1689 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1689 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1689.1709793907.6664884.submit parsl.executors.status_handling INFO: Allocated block ID 1690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1690 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1690 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1690.1709793907.66692.submit parsl.executors.status_handling INFO: Allocated block ID 1691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1691 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1691 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1691.1709793907.6673412.submit parsl.executors.status_handling INFO: Allocated block ID 1692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1692 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1692 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1692.1709793907.6677659.submit parsl.executors.status_handling INFO: Allocated block ID 1693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1693 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1693 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1693.1709793907.6682057.submit parsl.executors.status_handling INFO: Allocated block ID 1694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1694 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1694 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1694.1709793907.6686363.submit parsl.executors.status_handling INFO: Allocated block ID 1695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1695 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1695 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1695.1709793907.6690784.submit parsl.executors.status_handling INFO: Allocated block ID 1696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1696 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1696 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1696.1709793907.6695065.submit parsl.executors.status_handling INFO: Allocated block ID 1697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1697 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1697 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1697.1709793907.6699343.submit parsl.executors.status_handling INFO: Allocated block ID 1698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1698 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1698 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1698.1709793907.670357.submit parsl.executors.status_handling INFO: Allocated block ID 1699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1699 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1699 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1699.1709793907.6707785.submit parsl.executors.status_handling INFO: Allocated block ID 1700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1700 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1700 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1700.1709793907.6712031.submit parsl.executors.status_handling INFO: Allocated block ID 1701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1701 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1701 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1701.1709793907.6716225.submit parsl.executors.status_handling INFO: Allocated block ID 1702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1702 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1702 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1702.1709793907.672053.submit parsl.executors.status_handling INFO: Allocated block ID 1703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1703 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1703 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1703.1709793907.672472.submit parsl.executors.status_handling INFO: Allocated block ID 1704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1704 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1704 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1704.1709793907.6729045.submit parsl.executors.status_handling INFO: Allocated block ID 1705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1705 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1705 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1705.1709793907.673336.submit parsl.executors.status_handling INFO: Allocated block ID 1706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1706 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1706 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1706.1709793907.6737561.submit parsl.executors.status_handling INFO: Allocated block ID 1707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1707 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1707 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1707.1709793907.687329.submit parsl.executors.status_handling INFO: Allocated block ID 1708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1708 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1708 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1708.1709793907.6877787.submit parsl.executors.status_handling INFO: Allocated block ID 1709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1709 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1709 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1709.1709793907.688218.submit parsl.executors.status_handling INFO: Allocated block ID 1710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1710 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1710 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1710.1709793907.6886492.submit parsl.executors.status_handling INFO: Allocated block ID 1711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1711 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1711 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1711.1709793907.6890855.submit parsl.executors.status_handling INFO: Allocated block ID 1712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1712 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1712 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1712.1709793907.6895123.submit parsl.executors.status_handling INFO: Allocated block ID 1713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1713 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1713 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1713.1709793907.6899493.submit parsl.executors.status_handling INFO: Allocated block ID 1714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1714 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1714 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1714.1709793907.6903815.submit parsl.executors.status_handling INFO: Allocated block ID 1715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1715 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1715 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1715.1709793907.6908023.submit parsl.executors.status_handling INFO: Allocated block ID 1716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1716 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1716 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1716.1709793907.6912282.submit parsl.executors.status_handling INFO: Allocated block ID 1717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1717 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1717 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1717.1709793907.6916518.submit parsl.executors.status_handling INFO: Allocated block ID 1718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1718 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1718 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1718.1709793907.6920867.submit parsl.executors.status_handling INFO: Allocated block ID 1719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1719 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1719 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1719.1709793907.6925132.submit parsl.executors.status_handling INFO: Allocated block ID 1720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1720 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1720 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1720.1709793907.6929471.submit parsl.executors.status_handling INFO: Allocated block ID 1721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1721 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1721 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1721.1709793907.6933694.submit parsl.executors.status_handling INFO: Allocated block ID 1722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1722 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1722 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1722.1709793907.6937892.submit parsl.executors.status_handling INFO: Allocated block ID 1723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1723 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1723 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1723.1709793907.69422.submit parsl.executors.status_handling INFO: Allocated block ID 1724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1724 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1724 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1724.1709793907.6946428.submit parsl.executors.status_handling INFO: Allocated block ID 1725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1725 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1725 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1725.1709793907.69507.submit parsl.executors.status_handling INFO: Allocated block ID 1726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1726 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1726 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1726.1709793907.6954944.submit parsl.executors.status_handling INFO: Allocated block ID 1727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1727 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1727 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1727.1709793907.6959274.submit parsl.executors.status_handling INFO: Allocated block ID 1728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1728 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1728 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1728.1709793907.6963623.submit parsl.executors.status_handling INFO: Allocated block ID 1729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1729 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1729 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1729.1709793907.6967905.submit parsl.executors.status_handling INFO: Allocated block ID 1730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1730 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1730 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1730.1709793907.69723.submit parsl.executors.status_handling INFO: Allocated block ID 1731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1731 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1731 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1731.1709793907.6976528.submit parsl.executors.status_handling INFO: Allocated block ID 1732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1732 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1732 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1732.1709793907.6980863.submit parsl.executors.status_handling INFO: Allocated block ID 1733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1733 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1733 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1733.1709793907.698507.submit parsl.executors.status_handling INFO: Allocated block ID 1734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1734 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1734 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1734.1709793907.698936.submit parsl.executors.status_handling INFO: Allocated block ID 1735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1735 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1735 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1735.1709793907.6993682.submit parsl.executors.status_handling INFO: Allocated block ID 1736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1736 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1736 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1736.1709793907.6997936.submit parsl.executors.status_handling INFO: Allocated block ID 1737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1737 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1737 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1737.1709793907.7013607.submit parsl.executors.status_handling INFO: Allocated block ID 1738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1738 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1738 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1738.1709793907.7018151.submit parsl.executors.status_handling INFO: Allocated block ID 1739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1739 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1739 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1739.1709793907.7022552.submit parsl.executors.status_handling INFO: Allocated block ID 1740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1740 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1740 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1740.1709793907.7026784.submit parsl.executors.status_handling INFO: Allocated block ID 1741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1741 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1741 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1741.1709793907.7031198.submit parsl.executors.status_handling INFO: Allocated block ID 1742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1742 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1742 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1742.1709793907.703546.submit parsl.executors.status_handling INFO: Allocated block ID 1743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1743 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1743 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1743.1709793907.703976.submit parsl.executors.status_handling INFO: Allocated block ID 1744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1744 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1744 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1744.1709793907.704406.submit parsl.executors.status_handling INFO: Allocated block ID 1745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1745 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1745 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1745.1709793907.7048366.submit parsl.executors.status_handling INFO: Allocated block ID 1746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1746 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1746 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1746.1709793907.7052655.submit parsl.executors.status_handling INFO: Allocated block ID 1747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1747 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1747 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1747.1709793907.705685.submit parsl.executors.status_handling INFO: Allocated block ID 1748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1748 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1748 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1748.1709793907.7061107.submit parsl.executors.status_handling INFO: Allocated block ID 1749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1749 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1749 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1749.1709793907.7065318.submit parsl.executors.status_handling INFO: Allocated block ID 1750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1750 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1750 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1750.1709793907.706962.submit parsl.executors.status_handling INFO: Allocated block ID 1751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1751 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1751 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1751.1709793907.7073824.submit parsl.executors.status_handling INFO: Allocated block ID 1752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1752 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1752 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1752.1709793907.7078214.submit parsl.executors.status_handling INFO: Allocated block ID 1753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1753 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1753 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1753.1709793907.708253.submit parsl.executors.status_handling INFO: Allocated block ID 1754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1754 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1754 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1754.1709793907.7086732.submit parsl.executors.status_handling INFO: Allocated block ID 1755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1755 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1755 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1755.1709793907.7091045.submit parsl.executors.status_handling INFO: Allocated block ID 1756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1756 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1756 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1756.1709793907.7095299.submit parsl.executors.status_handling INFO: Allocated block ID 1757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1757 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1757 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1757.1709793907.7100718.submit parsl.executors.status_handling INFO: Allocated block ID 1758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1758 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1758 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1758.1709793907.7105162.submit parsl.executors.status_handling INFO: Allocated block ID 1759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1759 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1759 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1759.1709793907.7109447.submit parsl.executors.status_handling INFO: Allocated block ID 1760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1760 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1760 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1760.1709793907.7113671.submit parsl.executors.status_handling INFO: Allocated block ID 1761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1761 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1761 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1761.1709793907.7117877.submit parsl.executors.status_handling INFO: Allocated block ID 1762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1762 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1762 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1762.1709793907.7122219.submit parsl.executors.status_handling INFO: Allocated block ID 1763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1763 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1763 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1763.1709793907.7126431.submit parsl.executors.status_handling INFO: Allocated block ID 1764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1764 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1764 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1764.1709793907.713075.submit parsl.executors.status_handling INFO: Allocated block ID 1765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1765 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1765 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1765.1709793907.7135098.submit parsl.executors.status_handling INFO: Allocated block ID 1766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1766 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1766 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1766.1709793907.7139401.submit parsl.executors.status_handling INFO: Allocated block ID 1767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1767 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1767 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1767.1709793907.7143683.submit parsl.executors.status_handling INFO: Allocated block ID 1768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1768 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1768 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1768.1709793907.7147942.submit parsl.executors.status_handling INFO: Allocated block ID 1769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1769 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1769 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1769.1709793907.715224.submit parsl.executors.status_handling INFO: Allocated block ID 1770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1770 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1770 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1770.1709793907.715646.submit parsl.executors.status_handling INFO: Allocated block ID 1771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1771 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1771 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1771.1709793907.7160838.submit parsl.executors.status_handling INFO: Allocated block ID 1772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1772 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1772 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1772.1709793907.7165108.submit parsl.executors.status_handling INFO: Allocated block ID 1773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1773 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1773 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1773.1709793907.7169352.submit parsl.executors.status_handling INFO: Allocated block ID 1774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1774 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1774 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1774.1709793907.717358.submit parsl.executors.status_handling INFO: Allocated block ID 1775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1775 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1775 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1775.1709793907.7177806.submit parsl.executors.status_handling INFO: Allocated block ID 1776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1776 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1776 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1776.1709793907.7182071.submit parsl.executors.status_handling INFO: Allocated block ID 1777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1777 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1777 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1777.1709793907.7186365.submit parsl.executors.status_handling INFO: Allocated block ID 1778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1778 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1778 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1778.1709793907.719068.submit parsl.executors.status_handling INFO: Allocated block ID 1779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1779 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1779 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1779.1709793907.7194908.submit parsl.executors.status_handling INFO: Allocated block ID 1780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1780 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1780 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1780.1709793907.7199266.submit parsl.executors.status_handling INFO: Allocated block ID 1781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1781 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1781 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1781.1709793907.7203524.submit parsl.executors.status_handling INFO: Allocated block ID 1782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1782 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1782 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1782.1709793907.7207716.submit parsl.executors.status_handling INFO: Allocated block ID 1783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1783 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1783 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1783.1709793907.721329.submit parsl.executors.status_handling INFO: Allocated block ID 1784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1784 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1784 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1784.1709793907.721762.submit parsl.executors.status_handling INFO: Allocated block ID 1785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1785 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1785 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1785.1709793907.7221932.submit parsl.executors.status_handling INFO: Allocated block ID 1786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1786 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1786 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1786.1709793907.7226224.submit parsl.executors.status_handling INFO: Allocated block ID 1787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1787 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1787 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1787.1709793907.7230506.submit parsl.executors.status_handling INFO: Allocated block ID 1788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1788 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1788 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1788.1709793907.7234778.submit parsl.executors.status_handling INFO: Allocated block ID 1789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1789 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1789 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1789.1709793907.7239134.submit parsl.executors.status_handling INFO: Allocated block ID 1790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1790 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1790 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1790.1709793907.7243426.submit parsl.executors.status_handling INFO: Allocated block ID 1791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1791 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1791 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1791.1709793907.724763.submit parsl.executors.status_handling INFO: Allocated block ID 1792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1792 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1792 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1792.1709793907.7252064.submit parsl.executors.status_handling INFO: Allocated block ID 1793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1793 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1793 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1793.1709793907.7256362.submit parsl.executors.status_handling INFO: Allocated block ID 1794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1794 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1794 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1794.1709793907.7260692.submit parsl.executors.status_handling INFO: Allocated block ID 1795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1795 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1795 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1795.1709793907.7265003.submit parsl.executors.status_handling INFO: Allocated block ID 1796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1796 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1796 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1796.1709793907.7269363.submit parsl.executors.status_handling INFO: Allocated block ID 1797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1797 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1797 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1797.1709793907.7273595.submit parsl.executors.status_handling INFO: Allocated block ID 1798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1798 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1798 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1798.1709793907.7277806.submit parsl.executors.status_handling INFO: Allocated block ID 1799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1799 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1799 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1799.1709793907.7282057.submit parsl.executors.status_handling INFO: Allocated block ID 1800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1800 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1800 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1800.1709793907.728623.submit parsl.executors.status_handling INFO: Allocated block ID 1801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1801 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1801 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1801.1709793907.7290514.submit parsl.executors.status_handling INFO: Allocated block ID 1802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1802 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1802 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1802.1709793907.7294698.submit parsl.executors.status_handling INFO: Allocated block ID 1803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1803 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1803 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1803.1709793907.7298925.submit parsl.executors.status_handling INFO: Allocated block ID 1804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1804 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1804 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1804.1709793907.7303143.submit parsl.executors.status_handling INFO: Allocated block ID 1805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1805 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1805 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1805.1709793907.73073.submit parsl.executors.status_handling INFO: Allocated block ID 1806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1806 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1806 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1806.1709793907.731163.submit parsl.executors.status_handling INFO: Allocated block ID 1807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1807 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1807 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1807.1709793907.731585.submit parsl.executors.status_handling INFO: Allocated block ID 1808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1808 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1808 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1808.1709793907.7320085.submit parsl.executors.status_handling INFO: Allocated block ID 1809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1809 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1809 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1809.1709793907.7324257.submit parsl.executors.status_handling INFO: Allocated block ID 1810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1810 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1810 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1810.1709793907.732855.submit parsl.executors.status_handling INFO: Allocated block ID 1811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1811 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1811 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1811.1709793907.7332735.submit parsl.executors.status_handling INFO: Allocated block ID 1812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1812 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1812 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1812.1709793907.7336948.submit parsl.executors.status_handling INFO: Allocated block ID 1813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1813 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1813 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1813.1709793907.7341328.submit parsl.executors.status_handling INFO: Allocated block ID 1814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1814 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1814 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1814.1709793907.7345552.submit parsl.executors.status_handling INFO: Allocated block ID 1815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1815 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1815 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1815.1709793907.7349792.submit parsl.executors.status_handling INFO: Allocated block ID 1816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1816 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1816 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1816.1709793907.7354014.submit parsl.executors.status_handling INFO: Allocated block ID 1817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1817 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1817 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1817.1709793907.7358265.submit parsl.executors.status_handling INFO: Allocated block ID 1818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1818 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1818 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1818.1709793907.7362483.submit parsl.executors.status_handling INFO: Allocated block ID 1819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1819 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1819 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1819.1709793907.7366652.submit parsl.executors.status_handling INFO: Allocated block ID 1820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1820 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1820 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1820.1709793907.737089.submit parsl.executors.status_handling INFO: Allocated block ID 1821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1821 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1821 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1821.1709793907.737507.submit parsl.executors.status_handling INFO: Allocated block ID 1822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1822 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1822 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1822.1709793907.737932.submit parsl.executors.status_handling INFO: Allocated block ID 1823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1823 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1823 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1823.1709793907.73835.submit parsl.executors.status_handling INFO: Allocated block ID 1824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1824 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1824 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1824.1709793907.7387671.submit parsl.executors.status_handling INFO: Allocated block ID 1825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1825 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1825 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1825.1709793907.739198.submit parsl.executors.status_handling INFO: Allocated block ID 1826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1826 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1826 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1826.1709793907.7396204.submit parsl.executors.status_handling INFO: Allocated block ID 1827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1827 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1827 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1827.1709793907.7400527.submit parsl.executors.status_handling INFO: Allocated block ID 1828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1828 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1828 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1828.1709793907.740494.submit parsl.executors.status_handling INFO: Allocated block ID 1829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1829 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1829 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1829.1709793907.7409363.submit parsl.executors.status_handling INFO: Allocated block ID 1830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1830 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1830 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1830.1709793907.7413676.submit parsl.executors.status_handling INFO: Allocated block ID 1831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1831 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1831 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1831.1709793907.7418017.submit parsl.executors.status_handling INFO: Allocated block ID 1832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1832 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1832 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1832.1709793907.742228.submit parsl.executors.status_handling INFO: Allocated block ID 1833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1833 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1833 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1833.1709793907.742634.submit parsl.executors.status_handling INFO: Allocated block ID 1834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1834 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1834 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1834.1709793907.7430577.submit parsl.executors.status_handling INFO: Allocated block ID 1835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1835 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1835 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1835.1709793907.7434964.submit parsl.executors.status_handling INFO: Allocated block ID 1836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1836 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1836 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1836.1709793907.7442312.submit parsl.executors.status_handling INFO: Allocated block ID 1837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1837 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1837 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1837.1709793907.7446823.submit parsl.executors.status_handling INFO: Allocated block ID 1838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1838 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1838 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1838.1709793907.745115.submit parsl.executors.status_handling INFO: Allocated block ID 1839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1839 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1839 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1839.1709793907.7455237.submit parsl.executors.status_handling INFO: Allocated block ID 1840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1840 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1840 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1840.1709793907.7459319.submit parsl.executors.status_handling INFO: Allocated block ID 1841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1841 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1841 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1841.1709793907.7463322.submit parsl.executors.status_handling INFO: Allocated block ID 1842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1842 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1842 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1842.1709793907.7467315.submit parsl.executors.status_handling INFO: Allocated block ID 1843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1843 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1843 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1843.1709793907.7471414.submit parsl.executors.status_handling INFO: Allocated block ID 1844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1844 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1844 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1844.1709793907.7475393.submit parsl.executors.status_handling INFO: Allocated block ID 1845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1845 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1845 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1845.1709793907.747963.submit parsl.executors.status_handling INFO: Allocated block ID 1846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1846 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1846 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1846.1709793907.7483847.submit parsl.executors.status_handling INFO: Allocated block ID 1847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1847 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1847 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1847.1709793907.7488022.submit parsl.executors.status_handling INFO: Allocated block ID 1848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1848 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1848 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1848.1709793907.7492316.submit parsl.executors.status_handling INFO: Allocated block ID 1849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1849 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1849 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1849.1709793907.7496345.submit parsl.executors.status_handling INFO: Allocated block ID 1850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1850 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1850 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1850.1709793907.7500403.submit parsl.executors.status_handling INFO: Allocated block ID 1851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1851 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1851 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1851.1709793907.7504373.submit parsl.executors.status_handling INFO: Allocated block ID 1852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1852 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1852 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1852.1709793907.7508616.submit parsl.executors.status_handling INFO: Allocated block ID 1853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1853 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1853 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1853.1709793907.7512815.submit parsl.executors.status_handling INFO: Allocated block ID 1854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1854 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1854 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1854.1709793907.7516987.submit parsl.executors.status_handling INFO: Allocated block ID 1855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1855 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1855 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1855.1709793907.7521248.submit parsl.executors.status_handling INFO: Allocated block ID 1856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1856 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1856 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1856.1709793907.7525518.submit parsl.executors.status_handling INFO: Allocated block ID 1857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1857 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1857 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1857.1709793907.752961.submit parsl.executors.status_handling INFO: Allocated block ID 1858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1858 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1858 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1858.1709793907.753366.submit parsl.executors.status_handling INFO: Allocated block ID 1859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1859 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1859 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1859.1709793907.7537634.submit parsl.executors.status_handling INFO: Allocated block ID 1860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1860 2024-03-06 22:45:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1860 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1860.1709793907.7541668.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1356 parsl.dataflow.strategy DEBUG: Executor multi has 1356 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 1861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1861 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1861 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1861.1709793912.6393044.submit parsl.executors.status_handling INFO: Allocated block ID 1862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1862 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1862 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1862.1709793912.6398005.submit parsl.executors.status_handling INFO: Allocated block ID 1863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1863 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1863 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1863.1709793912.6402512.submit parsl.executors.status_handling INFO: Allocated block ID 1864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1864 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1864 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1864.1709793912.640678.submit parsl.executors.status_handling INFO: Allocated block ID 1865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1865 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1865 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1865.1709793912.641107.submit parsl.executors.status_handling INFO: Allocated block ID 1866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1866 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1866 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1866.1709793912.64153.submit parsl.executors.status_handling INFO: Allocated block ID 1867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1867 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1867 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1867.1709793912.6419587.submit parsl.executors.status_handling INFO: Allocated block ID 1868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1868 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1868 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1868.1709793912.6423864.submit parsl.executors.status_handling INFO: Allocated block ID 1869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1869 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1869 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1869.1709793912.6428125.submit parsl.executors.status_handling INFO: Allocated block ID 1870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1870 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1870 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1870.1709793912.6432402.submit parsl.executors.status_handling INFO: Allocated block ID 1871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1871 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1871 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1871.1709793912.6436632.submit parsl.executors.status_handling INFO: Allocated block ID 1872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1872 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1872 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1872.1709793912.6441052.submit parsl.executors.status_handling INFO: Allocated block ID 1873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1873 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1873 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1873.1709793912.644533.submit parsl.executors.status_handling INFO: Allocated block ID 1874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1874 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1874 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1874.1709793912.644968.submit parsl.executors.status_handling INFO: Allocated block ID 1875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1875 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1875 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1875.1709793912.6453886.submit parsl.executors.status_handling INFO: Allocated block ID 1876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1876 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1876 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1876.1709793912.645813.submit parsl.executors.status_handling INFO: Allocated block ID 1877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1877 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1877 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1877.1709793912.646243.submit parsl.executors.status_handling INFO: Allocated block ID 1878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1878 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1878 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1878.1709793912.6466699.submit parsl.executors.status_handling INFO: Allocated block ID 1879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1879 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1879 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1879.1709793912.6470983.submit parsl.executors.status_handling INFO: Allocated block ID 1880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1880 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1880 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1880.1709793912.6475246.submit parsl.executors.status_handling INFO: Allocated block ID 1881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1881 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1881 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1881.1709793912.647955.submit parsl.executors.status_handling INFO: Allocated block ID 1882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1882 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1882 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1882.1709793912.648378.submit parsl.executors.status_handling INFO: Allocated block ID 1883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1883 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1883 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1883.1709793912.648799.submit parsl.executors.status_handling INFO: Allocated block ID 1884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1884 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1884 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1884.1709793912.6492388.submit parsl.executors.status_handling INFO: Allocated block ID 1885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1885 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1885 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1885.1709793912.6496627.submit parsl.executors.status_handling INFO: Allocated block ID 1886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1886 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1886 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1886.1709793912.6500964.submit parsl.executors.status_handling INFO: Allocated block ID 1887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1887 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1887 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1887.1709793912.6505203.submit parsl.executors.status_handling INFO: Allocated block ID 1888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1888 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1888 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1888.1709793912.6509469.submit parsl.executors.status_handling INFO: Allocated block ID 1889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1889 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1889 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1889.1709793912.6513653.submit parsl.executors.status_handling INFO: Allocated block ID 1890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1890 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1890 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1890.1709793912.6517854.submit parsl.executors.status_handling INFO: Allocated block ID 1891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1891 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1891 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1891.1709793912.665349.submit parsl.executors.status_handling INFO: Allocated block ID 1892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1892 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1892 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1892.1709793912.6657834.submit parsl.executors.status_handling INFO: Allocated block ID 1893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1893 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1893 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1893.1709793912.666211.submit parsl.executors.status_handling INFO: Allocated block ID 1894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1894 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1894 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1894.1709793912.6666296.submit parsl.executors.status_handling INFO: Allocated block ID 1895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1895 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1895 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1895.1709793912.6670544.submit parsl.executors.status_handling INFO: Allocated block ID 1896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1896 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1896 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1896.1709793912.6674821.submit parsl.executors.status_handling INFO: Allocated block ID 1897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1897 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1897 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1897.1709793912.6679091.submit parsl.executors.status_handling INFO: Allocated block ID 1898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1898 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1898 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1898.1709793912.6683364.submit parsl.executors.status_handling INFO: Allocated block ID 1899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1899 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1899 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1899.1709793912.668752.submit parsl.executors.status_handling INFO: Allocated block ID 1900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1900 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1900 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1900.1709793912.6691778.submit parsl.executors.status_handling INFO: Allocated block ID 1901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1901 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1901 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1901.1709793912.6695957.submit parsl.executors.status_handling INFO: Allocated block ID 1902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1902 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1902 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1902.1709793912.6700196.submit parsl.executors.status_handling INFO: Allocated block ID 1903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1903 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1903 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1903.1709793912.6704466.submit parsl.executors.status_handling INFO: Allocated block ID 1904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1904 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1904 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1904.1709793912.6708806.submit parsl.executors.status_handling INFO: Allocated block ID 1905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1905 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1905 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1905.1709793912.6712997.submit parsl.executors.status_handling INFO: Allocated block ID 1906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1906 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1906 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1906.1709793912.6717193.submit parsl.executors.status_handling INFO: Allocated block ID 1907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1907 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1907 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1907.1709793912.672145.submit parsl.executors.status_handling INFO: Allocated block ID 1908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1908 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1908 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1908.1709793912.6725676.submit parsl.executors.status_handling INFO: Allocated block ID 1909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1909 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1909 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1909.1709793912.6729949.submit parsl.executors.status_handling INFO: Allocated block ID 1910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1910 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1910 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1910.1709793912.673421.submit parsl.executors.status_handling INFO: Allocated block ID 1911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1911 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1911 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1911.1709793912.6738424.submit parsl.executors.status_handling INFO: Allocated block ID 1912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1912 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1912 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1912.1709793912.6742654.submit parsl.executors.status_handling INFO: Allocated block ID 1913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1913 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1913 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1913.1709793912.6746836.submit parsl.executors.status_handling INFO: Allocated block ID 1914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1914 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1914 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1914.1709793912.6751137.submit parsl.executors.status_handling INFO: Allocated block ID 1915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1915 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1915 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1915.1709793912.675533.submit parsl.executors.status_handling INFO: Allocated block ID 1916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1916 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1916 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1916.1709793912.6759632.submit parsl.executors.status_handling INFO: Allocated block ID 1917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1917 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1917 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1917.1709793912.6763947.submit parsl.executors.status_handling INFO: Allocated block ID 1918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1918 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1918 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1918.1709793912.676822.submit parsl.executors.status_handling INFO: Allocated block ID 1919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1919 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1919 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1919.1709793912.6772423.submit parsl.executors.status_handling INFO: Allocated block ID 1920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1920 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1920 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1920.1709793912.6776745.submit parsl.executors.status_handling INFO: Allocated block ID 1921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1921 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1921 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1921.1709793912.6912487.submit parsl.executors.status_handling INFO: Allocated block ID 1922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1922 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1922 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1922.1709793912.691703.submit parsl.executors.status_handling INFO: Allocated block ID 1923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1923 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1923 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1923.1709793912.692157.submit parsl.executors.status_handling INFO: Allocated block ID 1924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1924 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1924 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1924.1709793912.6925876.submit parsl.executors.status_handling INFO: Allocated block ID 1925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1925 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1925 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1925.1709793912.6930356.submit parsl.executors.status_handling INFO: Allocated block ID 1926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1926 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1926 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1926.1709793912.693472.submit parsl.executors.status_handling INFO: Allocated block ID 1927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1927 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1927 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1927.1709793912.6938992.submit parsl.executors.status_handling INFO: Allocated block ID 1928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1928 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1928 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1928.1709793912.6943247.submit parsl.executors.status_handling INFO: Allocated block ID 1929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1929 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1929 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1929.1709793912.6947427.submit parsl.executors.status_handling INFO: Allocated block ID 1930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1930 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1930 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1930.1709793912.6951714.submit parsl.executors.status_handling INFO: Allocated block ID 1931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1931 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1931 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1931.1709793912.6955917.submit parsl.executors.status_handling INFO: Allocated block ID 1932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1932 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1932 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1932.1709793912.696033.submit parsl.executors.status_handling INFO: Allocated block ID 1933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1933 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1933 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1933.1709793912.6964557.submit parsl.executors.status_handling INFO: Allocated block ID 1934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1934 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1934 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1934.1709793912.6968844.submit parsl.executors.status_handling INFO: Allocated block ID 1935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1935 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1935 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1935.1709793912.6973064.submit parsl.executors.status_handling INFO: Allocated block ID 1936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1936 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1936 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1936.1709793912.69773.submit parsl.executors.status_handling INFO: Allocated block ID 1937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1937 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1937 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1937.1709793912.6981554.submit parsl.executors.status_handling INFO: Allocated block ID 1938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1938 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1938 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1938.1709793912.698575.submit parsl.executors.status_handling INFO: Allocated block ID 1939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1939 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1939 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1939.1709793912.699003.submit parsl.executors.status_handling INFO: Allocated block ID 1940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1940 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1940 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1940.1709793912.6994278.submit parsl.executors.status_handling INFO: Allocated block ID 1941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1941 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1941 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1941.1709793912.6998587.submit parsl.executors.status_handling INFO: Allocated block ID 1942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1942 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1942 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1942.1709793912.7002795.submit parsl.executors.status_handling INFO: Allocated block ID 1943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1943 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1943 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1943.1709793912.7006955.submit parsl.executors.status_handling INFO: Allocated block ID 1944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1944 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1944 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1944.1709793912.7011275.submit parsl.executors.status_handling INFO: Allocated block ID 1945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1945 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1945 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1945.1709793912.70155.submit parsl.executors.status_handling INFO: Allocated block ID 1946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1946 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1946 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1946.1709793912.7019804.submit parsl.executors.status_handling INFO: Allocated block ID 1947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1947 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1947 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1947.1709793912.7024102.submit parsl.executors.status_handling INFO: Allocated block ID 1948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1948 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1948 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1948.1709793912.7028403.submit parsl.executors.status_handling INFO: Allocated block ID 1949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1949 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1949 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1949.1709793912.703263.submit parsl.executors.status_handling INFO: Allocated block ID 1950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1950 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1950 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1950.1709793912.7036774.submit parsl.executors.status_handling INFO: Allocated block ID 1951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1951 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1951 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1951.1709793912.7172823.submit parsl.executors.status_handling INFO: Allocated block ID 1952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1952 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1952 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1952.1709793912.717759.submit parsl.executors.status_handling INFO: Allocated block ID 1953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1953 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1953 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1953.1709793912.7182162.submit parsl.executors.status_handling INFO: Allocated block ID 1954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1954 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1954 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1954.1709793912.7186382.submit parsl.executors.status_handling INFO: Allocated block ID 1955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1955 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1955 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1955.1709793912.7190633.submit parsl.executors.status_handling INFO: Allocated block ID 1956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1956 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1956 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1956.1709793912.719492.submit parsl.executors.status_handling INFO: Allocated block ID 1957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1957 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1957 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1957.1709793912.7199233.submit parsl.executors.status_handling INFO: Allocated block ID 1958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1958 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1958 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1958.1709793912.7203505.submit parsl.executors.status_handling INFO: Allocated block ID 1959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1959 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1959 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1959.1709793912.7207713.submit parsl.executors.status_handling INFO: Allocated block ID 1960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1960 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1960 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1960.1709793912.7212212.submit parsl.executors.status_handling INFO: Allocated block ID 1961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1961 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1961 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1961.1709793912.7216525.submit parsl.executors.status_handling INFO: Allocated block ID 1962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1962 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1962 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1962.1709793912.7221105.submit parsl.executors.status_handling INFO: Allocated block ID 1963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1963 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1963 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1963.1709793912.7225435.submit parsl.executors.status_handling INFO: Allocated block ID 1964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1964 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1964 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1964.1709793912.7229793.submit parsl.executors.status_handling INFO: Allocated block ID 1965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1965 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1965 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1965.1709793912.723402.submit parsl.executors.status_handling INFO: Allocated block ID 1966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1966 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1966 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1966.1709793912.7238295.submit parsl.executors.status_handling INFO: Allocated block ID 1967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1967 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1967 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1967.1709793912.724254.submit parsl.executors.status_handling INFO: Allocated block ID 1968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1968 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1968 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1968.1709793912.7246814.submit parsl.executors.status_handling INFO: Allocated block ID 1969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1969 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1969 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1969.1709793912.7251403.submit parsl.executors.status_handling INFO: Allocated block ID 1970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1970 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1970 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1970.1709793912.725577.submit parsl.executors.status_handling INFO: Allocated block ID 1971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1971 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1971 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1971.1709793912.7260263.submit parsl.executors.status_handling INFO: Allocated block ID 1972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1972 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1972 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1972.1709793912.7264466.submit parsl.executors.status_handling INFO: Allocated block ID 1973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1973 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1973 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1973.1709793912.7268713.submit parsl.executors.status_handling INFO: Allocated block ID 1974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1974 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1974 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1974.1709793912.7272916.submit parsl.executors.status_handling INFO: Allocated block ID 1975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1975 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1975 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1975.1709793912.7277088.submit parsl.executors.status_handling INFO: Allocated block ID 1976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1976 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1976 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1976.1709793912.7281356.submit parsl.executors.status_handling INFO: Allocated block ID 1977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1977 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1977 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1977.1709793912.7285612.submit parsl.executors.status_handling INFO: Allocated block ID 1978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1978 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1978 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1978.1709793912.7289903.submit parsl.executors.status_handling INFO: Allocated block ID 1979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1979 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1979 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1979.1709793912.7294087.submit parsl.executors.status_handling INFO: Allocated block ID 1980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1980 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1980 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1980.1709793912.7429647.submit parsl.executors.status_handling INFO: Allocated block ID 1981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1981 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1981 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1981.1709793912.7434926.submit parsl.executors.status_handling INFO: Allocated block ID 1982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1982 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1982 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1982.1709793912.7439373.submit parsl.executors.status_handling INFO: Allocated block ID 1983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1983 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1983 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1983.1709793912.7443745.submit parsl.executors.status_handling INFO: Allocated block ID 1984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1984 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1984 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1984.1709793912.7447972.submit parsl.executors.status_handling INFO: Allocated block ID 1985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1985 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1985 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1985.1709793912.7452295.submit parsl.executors.status_handling INFO: Allocated block ID 1986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1986 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1986 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1986.1709793912.7456527.submit parsl.executors.status_handling INFO: Allocated block ID 1987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1987 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1987 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1987.1709793912.7460766.submit parsl.executors.status_handling INFO: Allocated block ID 1988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1988 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1988 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1988.1709793912.746501.submit parsl.executors.status_handling INFO: Allocated block ID 1989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1989 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1989 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1989.1709793912.746929.submit parsl.executors.status_handling INFO: Allocated block ID 1990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1990 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1990 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1990.1709793912.747351.submit parsl.executors.status_handling INFO: Allocated block ID 1991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1991 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1991 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1991.1709793912.74777.submit parsl.executors.status_handling INFO: Allocated block ID 1992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1992 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1992 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1992.1709793912.7482138.submit parsl.executors.status_handling INFO: Allocated block ID 1993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1993 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1993 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1993.1709793912.7486408.submit parsl.executors.status_handling INFO: Allocated block ID 1994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1994 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1994 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1994.1709793912.7490737.submit parsl.executors.status_handling INFO: Allocated block ID 1995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1995 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1995 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1995.1709793912.7494957.submit parsl.executors.status_handling INFO: Allocated block ID 1996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1996 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1996 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1996.1709793912.7499259.submit parsl.executors.status_handling INFO: Allocated block ID 1997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1997 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1997 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1997.1709793912.7503476.submit parsl.executors.status_handling INFO: Allocated block ID 1998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1998 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1998 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1998.1709793912.7507646.submit parsl.executors.status_handling INFO: Allocated block ID 1999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1999 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1999 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=1999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-1999.1709793912.7511916.submit parsl.executors.status_handling INFO: Allocated block ID 2000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2000 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2000 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2000.1709793912.7516167.submit parsl.executors.status_handling INFO: Allocated block ID 2001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2001 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2001 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2001.1709793912.7520466.submit parsl.executors.status_handling INFO: Allocated block ID 2002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2002 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2002 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2002.1709793912.752468.submit parsl.executors.status_handling INFO: Allocated block ID 2003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2003 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2003 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2003.1709793912.752898.submit parsl.executors.status_handling INFO: Allocated block ID 2004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2004 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2004 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2004.1709793912.753328.submit parsl.executors.status_handling INFO: Allocated block ID 2005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2005 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2005 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2005.1709793912.7537572.submit parsl.executors.status_handling INFO: Allocated block ID 2006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2006 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2006 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2006.1709793912.7541904.submit parsl.executors.status_handling INFO: Allocated block ID 2007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2007 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2007 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2007.1709793912.7546167.submit parsl.executors.status_handling INFO: Allocated block ID 2008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2008 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2008 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2008.1709793912.7550447.submit parsl.executors.status_handling INFO: Allocated block ID 2009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2009 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2009 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2009.1709793912.7554648.submit parsl.executors.status_handling INFO: Allocated block ID 2010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2010 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2010 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2010.1709793912.7690158.submit parsl.executors.status_handling INFO: Allocated block ID 2011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2011 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2011 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2011.1709793912.7695305.submit parsl.executors.status_handling INFO: Allocated block ID 2012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2012 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2012 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2012.1709793912.7699726.submit parsl.executors.status_handling INFO: Allocated block ID 2013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2013 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2013 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2013.1709793912.7703972.submit parsl.executors.status_handling INFO: Allocated block ID 2014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2014 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2014 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2014.1709793912.7708266.submit parsl.executors.status_handling INFO: Allocated block ID 2015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2015 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2015 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2015.1709793912.771251.submit parsl.executors.status_handling INFO: Allocated block ID 2016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2016 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2016 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2016.1709793912.7716796.submit parsl.executors.status_handling INFO: Allocated block ID 2017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2017 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2017 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2017.1709793912.7721167.submit parsl.executors.status_handling INFO: Allocated block ID 2018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2018 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2018 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2018.1709793912.7725453.submit parsl.executors.status_handling INFO: Allocated block ID 2019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2019 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2019 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2019.1709793912.7729726.submit parsl.executors.status_handling INFO: Allocated block ID 2020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2020 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2020 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2020.1709793912.773393.submit parsl.executors.status_handling INFO: Allocated block ID 2021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2021 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2021 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2021.1709793912.7738192.submit parsl.executors.status_handling INFO: Allocated block ID 2022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2022 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2022 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2022.1709793912.774239.submit parsl.executors.status_handling INFO: Allocated block ID 2023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2023 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2023 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2023.1709793912.7746704.submit parsl.executors.status_handling INFO: Allocated block ID 2024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2024 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2024 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2024.1709793912.775103.submit parsl.executors.status_handling INFO: Allocated block ID 2025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2025 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2025 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2025.1709793912.7755294.submit parsl.executors.status_handling INFO: Allocated block ID 2026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2026 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2026 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2026.1709793912.7759693.submit parsl.executors.status_handling INFO: Allocated block ID 2027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2027 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2027 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2027.1709793912.7763922.submit parsl.executors.status_handling INFO: Allocated block ID 2028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2028 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2028 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2028.1709793912.776827.submit parsl.executors.status_handling INFO: Allocated block ID 2029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2029 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2029 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2029.1709793912.7772536.submit parsl.executors.status_handling INFO: Allocated block ID 2030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2030 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2030 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2030.1709793912.77768.submit parsl.executors.status_handling INFO: Allocated block ID 2031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2031 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2031 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2031.1709793912.778106.submit parsl.executors.status_handling INFO: Allocated block ID 2032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2032 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2032 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2032.1709793912.7785282.submit parsl.executors.status_handling INFO: Allocated block ID 2033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2033 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2033 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2033.1709793912.778956.submit parsl.executors.status_handling INFO: Allocated block ID 2034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2034 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2034 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2034.1709793912.77938.submit parsl.executors.status_handling INFO: Allocated block ID 2035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2035 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2035 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2035.1709793912.7797985.submit parsl.executors.status_handling INFO: Allocated block ID 2036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2036 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2036 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2036.1709793912.7802343.submit parsl.executors.status_handling INFO: Allocated block ID 2037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2037 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2037 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2037.1709793912.7806623.submit parsl.executors.status_handling INFO: Allocated block ID 2038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2038 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2038 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2038.1709793912.7810936.submit parsl.executors.status_handling INFO: Allocated block ID 2039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2039 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2039 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2039.1709793912.781516.submit parsl.executors.status_handling INFO: Allocated block ID 2040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2040 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2040 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2040.1709793912.7950559.submit parsl.executors.status_handling INFO: Allocated block ID 2041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2041 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2041 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2041.1709793912.7955415.submit parsl.executors.status_handling INFO: Allocated block ID 2042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2042 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2042 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2042.1709793912.795984.submit parsl.executors.status_handling INFO: Allocated block ID 2043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2043 2024-03-06 22:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2043 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2043.1709793912.79641.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b6812cbedba98a0a661bcd37052d0f8 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/febb4158-094b-4c79-94ef-9c0662918291_biasIsr_1123728_47.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/febb4158-094b-4c79-94ef-9c0662918291_biasIsr_1123728_47.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc9344b2ad1b094ce1d3494eac51255 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00de9571-108b-4d8f-a6fe-305288de808c_biasIsr_1123830_47.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/00de9571-108b-4d8f-a6fe-305288de808c_biasIsr_1123830_47.stderr 2024-03-06 22:45:17 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1354 parsl.dataflow.strategy DEBUG: Executor multi has 1354 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2044 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2044 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2044.1709793917.6934326.submit parsl.executors.status_handling INFO: Allocated block ID 2045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2045 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2045 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2045.1709793917.693988.submit parsl.executors.status_handling INFO: Allocated block ID 2046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2046 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2046 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2046.1709793917.6944451.submit parsl.executors.status_handling INFO: Allocated block ID 2047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2047 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2047 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2047.1709793917.6949332.submit parsl.executors.status_handling INFO: Allocated block ID 2048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2048 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2048 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2048.1709793917.6953807.submit parsl.executors.status_handling INFO: Allocated block ID 2049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2049 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2049 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2049.1709793917.695881.submit parsl.executors.status_handling INFO: Allocated block ID 2050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2050 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2050 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2050.1709793917.6963246.submit parsl.executors.status_handling INFO: Allocated block ID 2051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2051 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2051 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2051.1709793917.696757.submit parsl.executors.status_handling INFO: Allocated block ID 2052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2052 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2052 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2052.1709793917.697196.submit parsl.executors.status_handling INFO: Allocated block ID 2053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2053 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2053 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2053.1709793917.697623.submit parsl.executors.status_handling INFO: Allocated block ID 2054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2054 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2054 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2054.1709793917.6980567.submit parsl.executors.status_handling INFO: Allocated block ID 2055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2055 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2055 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2055.1709793917.6984837.submit parsl.executors.status_handling INFO: Allocated block ID 2056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2056 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2056 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2056.1709793917.698933.submit parsl.executors.status_handling INFO: Allocated block ID 2057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2057 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2057 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2057.1709793917.6993566.submit parsl.executors.status_handling INFO: Allocated block ID 2058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2058 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2058 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2058.1709793917.6997824.submit parsl.executors.status_handling INFO: Allocated block ID 2059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2059 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2059 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2059.1709793917.7004385.submit parsl.executors.status_handling INFO: Allocated block ID 2060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2060 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2060 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2060.1709793917.700913.submit parsl.executors.status_handling INFO: Allocated block ID 2061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2061 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2061 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2061.1709793917.7013574.submit parsl.executors.status_handling INFO: Allocated block ID 2062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2062 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2062 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2062.1709793917.701793.submit parsl.executors.status_handling INFO: Allocated block ID 2063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2063 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2063 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2063.1709793917.7022374.submit parsl.executors.status_handling INFO: Allocated block ID 2064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2064 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2064 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2064.1709793917.7026658.submit parsl.executors.status_handling INFO: Allocated block ID 2065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2065 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2065 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2065.1709793917.7031143.submit parsl.executors.status_handling INFO: Allocated block ID 2066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2066 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2066 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2066.1709793917.7035353.submit parsl.executors.status_handling INFO: Allocated block ID 2067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2067 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2067 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2067.1709793917.7039707.submit parsl.executors.status_handling INFO: Allocated block ID 2068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2068 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2068 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2068.1709793917.7044175.submit parsl.executors.status_handling INFO: Allocated block ID 2069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2069 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2069 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2069.1709793917.7048614.submit parsl.executors.status_handling INFO: Allocated block ID 2070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2070 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2070 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2070.1709793917.705298.submit parsl.executors.status_handling INFO: Allocated block ID 2071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2071 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2071 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2071.1709793917.705736.submit parsl.executors.status_handling INFO: Allocated block ID 2072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2072 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2072 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2072.1709793917.7061865.submit parsl.executors.status_handling INFO: Allocated block ID 2073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2073 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2073 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2073.1709793917.7066064.submit parsl.executors.status_handling INFO: Allocated block ID 2074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2074 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2074 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2074.1709793917.7070374.submit parsl.executors.status_handling INFO: Allocated block ID 2075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2075 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2075 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2075.1709793917.7074568.submit parsl.executors.status_handling INFO: Allocated block ID 2076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2076 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2076 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2076.1709793917.70788.submit parsl.executors.status_handling INFO: Allocated block ID 2077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2077 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2077 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2077.1709793917.7082982.submit parsl.executors.status_handling INFO: Allocated block ID 2078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2078 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2078 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2078.1709793917.708718.submit parsl.executors.status_handling INFO: Allocated block ID 2079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2079 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2079 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2079.1709793917.7091467.submit parsl.executors.status_handling INFO: Allocated block ID 2080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2080 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2080 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2080.1709793917.7095664.submit parsl.executors.status_handling INFO: Allocated block ID 2081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2081 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2081 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2081.1709793917.7099915.submit parsl.executors.status_handling INFO: Allocated block ID 2082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2082 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2082 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2082.1709793917.710412.submit parsl.executors.status_handling INFO: Allocated block ID 2083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2083 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2083 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2083.1709793917.7109911.submit parsl.executors.status_handling INFO: Allocated block ID 2084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2084 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2084 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2084.1709793917.7114394.submit parsl.executors.status_handling INFO: Allocated block ID 2085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2085 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2085 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2085.1709793917.7118888.submit parsl.executors.status_handling INFO: Allocated block ID 2086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2086 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2086 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2086.1709793917.7123275.submit parsl.executors.status_handling INFO: Allocated block ID 2087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2087 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2087 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2087.1709793917.7127457.submit parsl.executors.status_handling INFO: Allocated block ID 2088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2088 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2088 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2088.1709793917.713174.submit parsl.executors.status_handling INFO: Allocated block ID 2089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2089 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2089 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2089.1709793917.7136014.submit parsl.executors.status_handling INFO: Allocated block ID 2090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2090 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2090 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2090.1709793917.714026.submit parsl.executors.status_handling INFO: Allocated block ID 2091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2091 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2091 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2091.1709793917.7144434.submit parsl.executors.status_handling INFO: Allocated block ID 2092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2092 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2092 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2092.1709793917.714874.submit parsl.executors.status_handling INFO: Allocated block ID 2093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2093 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2093 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2093.1709793917.7152967.submit parsl.executors.status_handling INFO: Allocated block ID 2094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2094 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2094 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2094.1709793917.715713.submit parsl.executors.status_handling INFO: Allocated block ID 2095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2095 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2095 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2095.1709793917.7161424.submit parsl.executors.status_handling INFO: Allocated block ID 2096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2096 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2096 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2096.1709793917.7165632.submit parsl.executors.status_handling INFO: Allocated block ID 2097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2097 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2097 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2097.1709793917.7169945.submit parsl.executors.status_handling INFO: Allocated block ID 2098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2098 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2098 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2098.1709793917.7174172.submit parsl.executors.status_handling INFO: Allocated block ID 2099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2099 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2099 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2099.1709793917.7178423.submit parsl.executors.status_handling INFO: Allocated block ID 2100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2100 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2100 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2100.1709793917.7182631.submit parsl.executors.status_handling INFO: Allocated block ID 2101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2101 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2101 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2101.1709793917.7186837.submit parsl.executors.status_handling INFO: Allocated block ID 2102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2102 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2102 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2102.1709793917.7191105.submit parsl.executors.status_handling INFO: Allocated block ID 2103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2103 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2103 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2103.1709793917.719528.submit parsl.executors.status_handling INFO: Allocated block ID 2104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2104 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2104 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2104.1709793917.7199528.submit parsl.executors.status_handling INFO: Allocated block ID 2105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2105 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2105 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2105.1709793917.720372.submit parsl.executors.status_handling INFO: Allocated block ID 2106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2106 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2106 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2106.1709793917.7207894.submit parsl.executors.status_handling INFO: Allocated block ID 2107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2107 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2107 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2107.1709793917.7214606.submit parsl.executors.status_handling INFO: Allocated block ID 2108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2108 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2108 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2108.1709793917.7219112.submit parsl.executors.status_handling INFO: Allocated block ID 2109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2109 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2109 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2109.1709793917.7223396.submit parsl.executors.status_handling INFO: Allocated block ID 2110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2110 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2110 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2110.1709793917.7228937.submit parsl.executors.status_handling INFO: Allocated block ID 2111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2111 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2111 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2111.1709793917.7233324.submit parsl.executors.status_handling INFO: Allocated block ID 2112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2112 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2112 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2112.1709793917.7237558.submit parsl.executors.status_handling INFO: Allocated block ID 2113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2113 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2113 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2113.1709793917.7241907.submit parsl.executors.status_handling INFO: Allocated block ID 2114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2114 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2114 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2114.1709793917.7246156.submit parsl.executors.status_handling INFO: Allocated block ID 2115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2115 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2115 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2115.1709793917.7250624.submit parsl.executors.status_handling INFO: Allocated block ID 2116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2116 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2116 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2116.1709793917.7254956.submit parsl.executors.status_handling INFO: Allocated block ID 2117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2117 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2117 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2117.1709793917.7259362.submit parsl.executors.status_handling INFO: Allocated block ID 2118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2118 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2118 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2118.1709793917.7263737.submit parsl.executors.status_handling INFO: Allocated block ID 2119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2119 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2119 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2119.1709793917.726802.submit parsl.executors.status_handling INFO: Allocated block ID 2120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2120 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2120 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2120.1709793917.7272377.submit parsl.executors.status_handling INFO: Allocated block ID 2121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2121 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2121 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2121.1709793917.7276626.submit parsl.executors.status_handling INFO: Allocated block ID 2122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2122 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2122 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2122.1709793917.7280953.submit parsl.executors.status_handling INFO: Allocated block ID 2123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2123 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2123 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2123.1709793917.7285173.submit parsl.executors.status_handling INFO: Allocated block ID 2124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2124 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2124 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2124.1709793917.7289436.submit parsl.executors.status_handling INFO: Allocated block ID 2125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2125 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2125 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2125.1709793917.7293642.submit parsl.executors.status_handling INFO: Allocated block ID 2126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2126 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2126 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2126.1709793917.7297823.submit parsl.executors.status_handling INFO: Allocated block ID 2127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2127 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2127 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2127.1709793917.7302089.submit parsl.executors.status_handling INFO: Allocated block ID 2128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2128 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2128 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2128.1709793917.7306304.submit parsl.executors.status_handling INFO: Allocated block ID 2129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2129 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2129 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2129.1709793917.7310565.submit parsl.executors.status_handling INFO: Allocated block ID 2130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2130 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2130 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2130.1709793917.7314837.submit parsl.executors.status_handling INFO: Allocated block ID 2131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2131 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2131 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2131.1709793917.7320037.submit parsl.executors.status_handling INFO: Allocated block ID 2132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2132 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2132 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2132.1709793917.7324438.submit parsl.executors.status_handling INFO: Allocated block ID 2133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2133 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2133 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2133.1709793917.7328868.submit parsl.executors.status_handling INFO: Allocated block ID 2134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2134 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2134 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2134.1709793917.7333186.submit parsl.executors.status_handling INFO: Allocated block ID 2135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2135 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2135 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2135.1709793917.7337418.submit parsl.executors.status_handling INFO: Allocated block ID 2136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2136 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2136 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2136.1709793917.7341678.submit parsl.executors.status_handling INFO: Allocated block ID 2137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2137 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2137 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2137.1709793917.734589.submit parsl.executors.status_handling INFO: Allocated block ID 2138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2138 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2138 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2138.1709793917.735017.submit parsl.executors.status_handling INFO: Allocated block ID 2139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2139 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2139 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2139.1709793917.735436.submit parsl.executors.status_handling INFO: Allocated block ID 2140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2140 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2140 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2140.1709793917.7358646.submit parsl.executors.status_handling INFO: Allocated block ID 2141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2141 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2141 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2141.1709793917.736283.submit parsl.executors.status_handling INFO: Allocated block ID 2142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2142 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2142 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2142.1709793917.7366996.submit parsl.executors.status_handling INFO: Allocated block ID 2143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2143 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2143 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2143.1709793917.7371356.submit parsl.executors.status_handling INFO: Allocated block ID 2144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2144 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2144 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2144.1709793917.737557.submit parsl.executors.status_handling INFO: Allocated block ID 2145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2145 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2145 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2145.1709793917.7379842.submit parsl.executors.status_handling INFO: Allocated block ID 2146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2146 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2146 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2146.1709793917.7384067.submit parsl.executors.status_handling INFO: Allocated block ID 2147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2147 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2147 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2147.1709793917.7388341.submit parsl.executors.status_handling INFO: Allocated block ID 2148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2148 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2148 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2148.1709793917.7392569.submit parsl.executors.status_handling INFO: Allocated block ID 2149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2149 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2149 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2149.1709793917.739681.submit parsl.executors.status_handling INFO: Allocated block ID 2150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2150 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2150 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2150.1709793917.7401052.submit parsl.executors.status_handling INFO: Allocated block ID 2151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2151 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2151 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2151.1709793917.7405236.submit parsl.executors.status_handling INFO: Allocated block ID 2152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2152 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2152 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2152.1709793917.7409537.submit parsl.executors.status_handling INFO: Allocated block ID 2153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2153 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2153 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2153.1709793917.7413735.submit parsl.executors.status_handling INFO: Allocated block ID 2154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2154 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2154 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2154.1709793917.7417953.submit parsl.executors.status_handling INFO: Allocated block ID 2155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2155 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2155 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2155.1709793917.7422357.submit parsl.executors.status_handling INFO: Allocated block ID 2156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2156 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2156 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2156.1709793917.7426615.submit parsl.executors.status_handling INFO: Allocated block ID 2157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2157 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2157 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2157.1709793917.7431867.submit parsl.executors.status_handling INFO: Allocated block ID 2158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2158 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2158 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2158.1709793917.7436202.submit parsl.executors.status_handling INFO: Allocated block ID 2159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2159 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2159 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2159.1709793917.744054.submit parsl.executors.status_handling INFO: Allocated block ID 2160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2160 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2160 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2160.1709793917.7444804.submit parsl.executors.status_handling INFO: Allocated block ID 2161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2161 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2161 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2161.1709793917.7449095.submit parsl.executors.status_handling INFO: Allocated block ID 2162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2162 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2162 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2162.1709793917.7453327.submit parsl.executors.status_handling INFO: Allocated block ID 2163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2163 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2163 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2163.1709793917.7457554.submit parsl.executors.status_handling INFO: Allocated block ID 2164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2164 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2164 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2164.1709793917.7461889.submit parsl.executors.status_handling INFO: Allocated block ID 2165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2165 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2165 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2165.1709793917.746612.submit parsl.executors.status_handling INFO: Allocated block ID 2166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2166 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2166 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2166.1709793917.7470434.submit parsl.executors.status_handling INFO: Allocated block ID 2167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2167 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2167 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2167.1709793917.7474687.submit parsl.executors.status_handling INFO: Allocated block ID 2168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2168 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2168 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2168.1709793917.7478971.submit parsl.executors.status_handling INFO: Allocated block ID 2169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2169 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2169 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2169.1709793917.748318.submit parsl.executors.status_handling INFO: Allocated block ID 2170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2170 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2170 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2170.1709793917.748743.submit parsl.executors.status_handling INFO: Allocated block ID 2171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2171 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2171 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2171.1709793917.7491696.submit parsl.executors.status_handling INFO: Allocated block ID 2172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2172 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2172 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2172.1709793917.7495992.submit parsl.executors.status_handling INFO: Allocated block ID 2173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2173 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2173 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2173.1709793917.7500257.submit parsl.executors.status_handling INFO: Allocated block ID 2174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2174 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2174 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2174.1709793917.7504525.submit parsl.executors.status_handling INFO: Allocated block ID 2175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2175 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2175 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2175.1709793917.750888.submit parsl.executors.status_handling INFO: Allocated block ID 2176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2176 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2176 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2176.1709793917.7513173.submit parsl.executors.status_handling INFO: Allocated block ID 2177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2177 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2177 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2177.1709793917.751737.submit parsl.executors.status_handling INFO: Allocated block ID 2178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2178 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2178 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2178.1709793917.7521644.submit parsl.executors.status_handling INFO: Allocated block ID 2179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2179 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2179 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2179.1709793917.752594.submit parsl.executors.status_handling INFO: Allocated block ID 2180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2180 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2180 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2180.1709793917.7530246.submit parsl.executors.status_handling INFO: Allocated block ID 2181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2181 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2181 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2181.1709793917.7534459.submit parsl.executors.status_handling INFO: Allocated block ID 2182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2182 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2182 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2182.1709793917.7539608.submit parsl.executors.status_handling INFO: Allocated block ID 2183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2183 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2183 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2183.1709793917.7543988.submit parsl.executors.status_handling INFO: Allocated block ID 2184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2184 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2184 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2184.1709793917.7548313.submit parsl.executors.status_handling INFO: Allocated block ID 2185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2185 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2185 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2185.1709793917.755252.submit parsl.executors.status_handling INFO: Allocated block ID 2186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2186 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2186 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2186.1709793917.7556722.submit parsl.executors.status_handling INFO: Allocated block ID 2187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2187 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2187 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2187.1709793917.756105.submit parsl.executors.status_handling INFO: Allocated block ID 2188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2188 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2188 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2188.1709793917.7565298.submit parsl.executors.status_handling INFO: Allocated block ID 2189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2189 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2189 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2189.1709793917.7569616.submit parsl.executors.status_handling INFO: Allocated block ID 2190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2190 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2190 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2190.1709793917.75738.submit parsl.executors.status_handling INFO: Allocated block ID 2191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2191 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2191 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2191.1709793917.7578127.submit parsl.executors.status_handling INFO: Allocated block ID 2192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2192 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2192 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2192.1709793917.75824.submit parsl.executors.status_handling INFO: Allocated block ID 2193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2193 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2193 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2193.1709793917.7586608.submit parsl.executors.status_handling INFO: Allocated block ID 2194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2194 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2194 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2194.1709793917.7590964.submit parsl.executors.status_handling INFO: Allocated block ID 2195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2195 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2195 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2195.1709793917.7595184.submit parsl.executors.status_handling INFO: Allocated block ID 2196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2196 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2196 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2196.1709793917.759951.submit parsl.executors.status_handling INFO: Allocated block ID 2197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2197 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2197 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2197.1709793917.7603722.submit parsl.executors.status_handling INFO: Allocated block ID 2198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2198 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2198 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2198.1709793917.7607944.submit parsl.executors.status_handling INFO: Allocated block ID 2199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2199 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2199 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2199.1709793917.761219.submit parsl.executors.status_handling INFO: Allocated block ID 2200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2200 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2200 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2200.1709793917.7616396.submit parsl.executors.status_handling INFO: Allocated block ID 2201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2201 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2201 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2201.1709793917.7620666.submit parsl.executors.status_handling INFO: Allocated block ID 2202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2202 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2202 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2202.1709793917.7624884.submit parsl.executors.status_handling INFO: Allocated block ID 2203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2203 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2203 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2203.1709793917.7629235.submit parsl.executors.status_handling INFO: Allocated block ID 2204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2204 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2204 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2204.1709793917.7633464.submit parsl.executors.status_handling INFO: Allocated block ID 2205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2205 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2205 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2205.1709793917.7637675.submit parsl.executors.status_handling INFO: Allocated block ID 2206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2206 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2206 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2206.1709793917.764205.submit parsl.executors.status_handling INFO: Allocated block ID 2207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2207 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2207 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2207.1709793917.7646236.submit parsl.executors.status_handling INFO: Allocated block ID 2208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2208 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2208 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2208.1709793917.7651339.submit parsl.executors.status_handling INFO: Allocated block ID 2209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2209 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2209 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2209.1709793917.765565.submit parsl.executors.status_handling INFO: Allocated block ID 2210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2210 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2210 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2210.1709793917.7659934.submit parsl.executors.status_handling INFO: Allocated block ID 2211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2211 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2211 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2211.1709793917.7664125.submit parsl.executors.status_handling INFO: Allocated block ID 2212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2212 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2212 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2212.1709793917.7668424.submit parsl.executors.status_handling INFO: Allocated block ID 2213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2213 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2213 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2213.1709793917.7672648.submit parsl.executors.status_handling INFO: Allocated block ID 2214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2214 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2214 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2214.1709793917.7676854.submit parsl.executors.status_handling INFO: Allocated block ID 2215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2215 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2215 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2215.1709793917.7681162.submit parsl.executors.status_handling INFO: Allocated block ID 2216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2216 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2216 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2216.1709793917.7685406.submit parsl.executors.status_handling INFO: Allocated block ID 2217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2217 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2217 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2217.1709793917.7689776.submit parsl.executors.status_handling INFO: Allocated block ID 2218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2218 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2218 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2218.1709793917.7694087.submit parsl.executors.status_handling INFO: Allocated block ID 2219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2219 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2219 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2219.1709793917.7698338.submit parsl.executors.status_handling INFO: Allocated block ID 2220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2220 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2220 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2220.1709793917.7702568.submit parsl.executors.status_handling INFO: Allocated block ID 2221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2221 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2221 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2221.1709793917.7706764.submit parsl.executors.status_handling INFO: Allocated block ID 2222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2222 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2222 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2222.1709793917.7711062.submit parsl.executors.status_handling INFO: Allocated block ID 2223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2223 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2223 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2223.1709793917.771525.submit parsl.executors.status_handling INFO: Allocated block ID 2224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2224 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2224 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2224.1709793917.7719533.submit parsl.executors.status_handling INFO: Allocated block ID 2225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2225 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2225 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2225.1709793917.772374.submit parsl.executors.status_handling INFO: Allocated block ID 2226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2226 2024-03-06 22:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2226 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2226.1709793917.7727919.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1354 parsl.dataflow.strategy DEBUG: Executor multi has 1354 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2227 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2227 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2227.1709793922.6495547.submit parsl.executors.status_handling INFO: Allocated block ID 2228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2228 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2228 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2228.1709793922.6516361.submit parsl.executors.status_handling INFO: Allocated block ID 2229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2229 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2229 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2229.1709793922.653324.submit parsl.executors.status_handling INFO: Allocated block ID 2230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2230 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2230 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2230.1709793922.6541965.submit parsl.executors.status_handling INFO: Allocated block ID 2231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2231 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2231 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2231.1709793922.6550899.submit parsl.executors.status_handling INFO: Allocated block ID 2232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2232 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2232 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2232.1709793922.656167.submit parsl.executors.status_handling INFO: Allocated block ID 2233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2233 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2233 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2233.1709793922.6570716.submit parsl.executors.status_handling INFO: Allocated block ID 2234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2234 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2234 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2234.1709793922.6579533.submit parsl.executors.status_handling INFO: Allocated block ID 2235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2235 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2235 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2235.1709793922.6588175.submit parsl.executors.status_handling INFO: Allocated block ID 2236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2236 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2236 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2236.1709793922.6596792.submit parsl.executors.status_handling INFO: Allocated block ID 2237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2237 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2237 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2237.1709793922.6605494.submit parsl.executors.status_handling INFO: Allocated block ID 2238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2238 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2238 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2238.1709793922.6614432.submit parsl.executors.status_handling INFO: Allocated block ID 2239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2239 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2239 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2239.1709793922.6623144.submit parsl.executors.status_handling INFO: Allocated block ID 2240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2240 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2240 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2240.1709793922.6631885.submit parsl.executors.status_handling INFO: Allocated block ID 2241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2241 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2241 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2241.1709793922.663608.submit parsl.executors.status_handling INFO: Allocated block ID 2242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2242 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2242 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2242.1709793922.664033.submit parsl.executors.status_handling INFO: Allocated block ID 2243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2243 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2243 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2243.1709793922.6644526.submit parsl.executors.status_handling INFO: Allocated block ID 2244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2244 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2244 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2244.1709793922.6648798.submit parsl.executors.status_handling INFO: Allocated block ID 2245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2245 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2245 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2245.1709793922.6652975.submit parsl.executors.status_handling INFO: Allocated block ID 2246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2246 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2246 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2246.1709793922.6657155.submit parsl.executors.status_handling INFO: Allocated block ID 2247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2247 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2247 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2247.1709793922.6662903.submit parsl.executors.status_handling INFO: Allocated block ID 2248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2248 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2248 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2248.1709793922.6667316.submit parsl.executors.status_handling INFO: Allocated block ID 2249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2249 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2249 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2249.1709793922.667163.submit parsl.executors.status_handling INFO: Allocated block ID 2250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2250 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2250 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2250.1709793922.6675882.submit parsl.executors.status_handling INFO: Allocated block ID 2251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2251 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2251 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2251.1709793922.6680179.submit parsl.executors.status_handling INFO: Allocated block ID 2252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2252 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2252 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2252.1709793922.6684432.submit parsl.executors.status_handling INFO: Allocated block ID 2253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2253 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2253 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2253.1709793922.6688647.submit parsl.executors.status_handling INFO: Allocated block ID 2254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2254 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2254 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2254.1709793922.6692832.submit parsl.executors.status_handling INFO: Allocated block ID 2255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2255 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2255 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2255.1709793922.6697018.submit parsl.executors.status_handling INFO: Allocated block ID 2256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2256 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2256 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2256.1709793922.6701233.submit parsl.executors.status_handling INFO: Allocated block ID 2257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2257 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2257 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2257.1709793922.6705434.submit parsl.executors.status_handling INFO: Allocated block ID 2258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2258 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2258 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2258.1709793922.6709676.submit parsl.executors.status_handling INFO: Allocated block ID 2259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2259 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2259 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2259.1709793922.6713858.submit parsl.executors.status_handling INFO: Allocated block ID 2260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2260 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2260 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2260.1709793922.6718147.submit parsl.executors.status_handling INFO: Allocated block ID 2261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2261 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2261 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2261.1709793922.6722355.submit parsl.executors.status_handling INFO: Allocated block ID 2262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2262 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2262 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2262.1709793922.672665.submit parsl.executors.status_handling INFO: Allocated block ID 2263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2263 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2263 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2263.1709793922.6730974.submit parsl.executors.status_handling INFO: Allocated block ID 2264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2264 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2264 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2264.1709793922.6735225.submit parsl.executors.status_handling INFO: Allocated block ID 2265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2265 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2265 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2265.1709793922.6739466.submit parsl.executors.status_handling INFO: Allocated block ID 2266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2266 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2266 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2266.1709793922.6743643.submit parsl.executors.status_handling INFO: Allocated block ID 2267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2267 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2267 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2267.1709793922.674779.submit parsl.executors.status_handling INFO: Allocated block ID 2268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2268 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2268 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2268.1709793922.6752055.submit parsl.executors.status_handling INFO: Allocated block ID 2269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2269 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2269 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2269.1709793922.6756217.submit parsl.executors.status_handling INFO: Allocated block ID 2270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2270 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2270 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2270.1709793922.6773593.submit parsl.executors.status_handling INFO: Allocated block ID 2271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2271 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2271 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2271.1709793922.6778457.submit parsl.executors.status_handling INFO: Allocated block ID 2272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2272 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2272 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2272.1709793922.6782928.submit parsl.executors.status_handling INFO: Allocated block ID 2273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2273 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2273 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2273.1709793922.6787298.submit parsl.executors.status_handling INFO: Allocated block ID 2274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2274 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2274 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2274.1709793922.67919.submit parsl.executors.status_handling INFO: Allocated block ID 2275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2275 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2275 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2275.1709793922.6796124.submit parsl.executors.status_handling INFO: Allocated block ID 2276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2276 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2276 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2276.1709793922.680045.submit parsl.executors.status_handling INFO: Allocated block ID 2277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2277 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2277 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2277.1709793922.6804702.submit parsl.executors.status_handling INFO: Allocated block ID 2278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2278 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2278 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2278.1709793922.680895.submit parsl.executors.status_handling INFO: Allocated block ID 2279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2279 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2279 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2279.1709793922.6813169.submit parsl.executors.status_handling INFO: Allocated block ID 2280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2280 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2280 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2280.1709793922.6817374.submit parsl.executors.status_handling INFO: Allocated block ID 2281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2281 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2281 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2281.1709793922.6821644.submit parsl.executors.status_handling INFO: Allocated block ID 2282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2282 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2282 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2282.1709793922.6825886.submit parsl.executors.status_handling INFO: Allocated block ID 2283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2283 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2283 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2283.1709793922.6830106.submit parsl.executors.status_handling INFO: Allocated block ID 2284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2284 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2284 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2284.1709793922.6834252.submit parsl.executors.status_handling INFO: Allocated block ID 2285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2285 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2285 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2285.1709793922.6838496.submit parsl.executors.status_handling INFO: Allocated block ID 2286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2286 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2286 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2286.1709793922.6842754.submit parsl.executors.status_handling INFO: Allocated block ID 2287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2287 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2287 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2287.1709793922.6846955.submit parsl.executors.status_handling INFO: Allocated block ID 2288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2288 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2288 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2288.1709793922.6851277.submit parsl.executors.status_handling INFO: Allocated block ID 2289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2289 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2289 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2289.1709793922.6855457.submit parsl.executors.status_handling INFO: Allocated block ID 2290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2290 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2290 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2290.1709793922.6859899.submit parsl.executors.status_handling INFO: Allocated block ID 2291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2291 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2291 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2291.1709793922.6866019.submit parsl.executors.status_handling INFO: Allocated block ID 2292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2292 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2292 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2292.1709793922.6870556.submit parsl.executors.status_handling INFO: Allocated block ID 2293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2293 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2293 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2293.1709793922.6874998.submit parsl.executors.status_handling INFO: Allocated block ID 2294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2294 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2294 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2294.1709793922.687937.submit parsl.executors.status_handling INFO: Allocated block ID 2295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2295 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2295 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2295.1709793922.6883612.submit parsl.executors.status_handling INFO: Allocated block ID 2296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2296 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2296 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2296.1709793922.688804.submit parsl.executors.status_handling INFO: Allocated block ID 2297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2297 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2297 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2297.1709793922.6892426.submit parsl.executors.status_handling INFO: Allocated block ID 2298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2298 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2298 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2298.1709793922.6896896.submit parsl.executors.status_handling INFO: Allocated block ID 2299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2299 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2299 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2299.1709793922.690124.submit parsl.executors.status_handling INFO: Allocated block ID 2300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2300 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2300 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2300.1709793922.6905463.submit parsl.executors.status_handling INFO: Allocated block ID 2301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2301 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2301 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2301.1709793922.6909678.submit parsl.executors.status_handling INFO: Allocated block ID 2302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2302 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2302 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2302.1709793922.6913943.submit parsl.executors.status_handling INFO: Allocated block ID 2303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2303 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2303 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2303.1709793922.6918232.submit parsl.executors.status_handling INFO: Allocated block ID 2304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2304 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2304 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2304.1709793922.6922457.submit parsl.executors.status_handling INFO: Allocated block ID 2305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2305 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2305 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2305.1709793922.6926665.submit parsl.executors.status_handling INFO: Allocated block ID 2306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2306 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2306 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2306.1709793922.6930919.submit parsl.executors.status_handling INFO: Allocated block ID 2307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2307 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2307 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2307.1709793922.6935184.submit parsl.executors.status_handling INFO: Allocated block ID 2308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2308 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2308 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2308.1709793922.693943.submit parsl.executors.status_handling INFO: Allocated block ID 2309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2309 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2309 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2309.1709793922.694363.submit parsl.executors.status_handling INFO: Allocated block ID 2310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2310 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2310 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2310.1709793922.694789.submit parsl.executors.status_handling INFO: Allocated block ID 2311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2311 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2311 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2311.1709793922.6952174.submit parsl.executors.status_handling INFO: Allocated block ID 2312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2312 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2312 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2312.1709793922.695641.submit parsl.executors.status_handling INFO: Allocated block ID 2313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2313 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2313 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2313.1709793922.6960685.submit parsl.executors.status_handling INFO: Allocated block ID 2314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2314 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2314 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2314.1709793922.6966183.submit parsl.executors.status_handling INFO: Allocated block ID 2315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2315 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2315 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2315.1709793922.6970537.submit parsl.executors.status_handling INFO: Allocated block ID 2316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2316 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2316 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2316.1709793922.6974726.submit parsl.executors.status_handling INFO: Allocated block ID 2317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2317 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2317 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2317.1709793922.6978974.submit parsl.executors.status_handling INFO: Allocated block ID 2318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2318 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2318 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2318.1709793922.69832.submit parsl.executors.status_handling INFO: Allocated block ID 2319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2319 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2319 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2319.1709793922.6987367.submit parsl.executors.status_handling INFO: Allocated block ID 2320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2320 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2320 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2320.1709793922.6991615.submit parsl.executors.status_handling INFO: Allocated block ID 2321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2321 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2321 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2321.1709793922.69958.submit parsl.executors.status_handling INFO: Allocated block ID 2322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2322 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2322 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2322.1709793922.7000115.submit parsl.executors.status_handling INFO: Allocated block ID 2323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2323 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2323 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2323.1709793922.700444.submit parsl.executors.status_handling INFO: Allocated block ID 2324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2324 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2324 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2324.1709793922.7008758.submit parsl.executors.status_handling INFO: Allocated block ID 2325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2325 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2325 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2325.1709793922.701295.submit parsl.executors.status_handling INFO: Allocated block ID 2326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2326 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2326 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2326.1709793922.7017095.submit parsl.executors.status_handling INFO: Allocated block ID 2327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2327 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2327 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2327.1709793922.7021382.submit parsl.executors.status_handling INFO: Allocated block ID 2328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2328 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2328 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2328.1709793922.7025554.submit parsl.executors.status_handling INFO: Allocated block ID 2329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2329 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2329 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2329.1709793922.7029746.submit parsl.executors.status_handling INFO: Allocated block ID 2330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2330 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2330 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2330.1709793922.7033958.submit parsl.executors.status_handling INFO: Allocated block ID 2331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2331 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2331 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2331.1709793922.7038229.submit parsl.executors.status_handling INFO: Allocated block ID 2332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2332 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2332 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2332.1709793922.7042406.submit parsl.executors.status_handling INFO: Allocated block ID 2333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2333 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2333 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2333.1709793922.704662.submit parsl.executors.status_handling INFO: Allocated block ID 2334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2334 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2334 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2334.1709793922.7050955.submit parsl.executors.status_handling INFO: Allocated block ID 2335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2335 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2335 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2335.1709793922.705516.submit parsl.executors.status_handling INFO: Allocated block ID 2336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2336 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2336 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2336.1709793922.7059467.submit parsl.executors.status_handling INFO: Allocated block ID 2337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2337 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2337 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2337.1709793922.7151175.submit parsl.executors.status_handling INFO: Allocated block ID 2338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2338 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2338 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2338.1709793922.7155783.submit parsl.executors.status_handling INFO: Allocated block ID 2339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2339 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2339 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2339.1709793922.7160177.submit parsl.executors.status_handling INFO: Allocated block ID 2340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2340 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2340 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2340.1709793922.7164805.submit parsl.executors.status_handling INFO: Allocated block ID 2341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2341 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2341 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2341.1709793922.7169206.submit parsl.executors.status_handling INFO: Allocated block ID 2342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2342 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2342 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2342.1709793922.7173486.submit parsl.executors.status_handling INFO: Allocated block ID 2343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2343 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2343 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2343.1709793922.7177725.submit parsl.executors.status_handling INFO: Allocated block ID 2344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2344 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2344 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2344.1709793922.7182026.submit parsl.executors.status_handling INFO: Allocated block ID 2345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2345 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2345 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2345.1709793922.7186255.submit parsl.executors.status_handling INFO: Allocated block ID 2346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2346 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2346 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2346.1709793922.7190573.submit parsl.executors.status_handling INFO: Allocated block ID 2347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2347 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2347 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2347.1709793922.719479.submit parsl.executors.status_handling INFO: Allocated block ID 2348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2348 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2348 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2348.1709793922.7199078.submit parsl.executors.status_handling INFO: Allocated block ID 2349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2349 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2349 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2349.1709793922.7203302.submit parsl.executors.status_handling INFO: Allocated block ID 2350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2350 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2350 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2350.1709793922.7207499.submit parsl.executors.status_handling INFO: Allocated block ID 2351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2351 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2351 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2351.1709793922.7211773.submit parsl.executors.status_handling INFO: Allocated block ID 2352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2352 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2352 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2352.1709793922.721603.submit parsl.executors.status_handling INFO: Allocated block ID 2353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2353 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2353 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2353.1709793922.7220297.submit parsl.executors.status_handling INFO: Allocated block ID 2354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2354 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2354 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2354.1709793922.7224517.submit parsl.executors.status_handling INFO: Allocated block ID 2355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2355 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2355 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2355.1709793922.722879.submit parsl.executors.status_handling INFO: Allocated block ID 2356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2356 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2356 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2356.1709793922.723301.submit parsl.executors.status_handling INFO: Allocated block ID 2357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2357 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2357 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2357.1709793922.7237191.submit parsl.executors.status_handling INFO: Allocated block ID 2358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2358 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2358 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2358.1709793922.7241554.submit parsl.executors.status_handling INFO: Allocated block ID 2359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2359 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2359 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2359.1709793922.72458.submit parsl.executors.status_handling INFO: Allocated block ID 2360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2360 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2360 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2360.1709793922.7339964.submit parsl.executors.status_handling INFO: Allocated block ID 2361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2361 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2361 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2361.1709793922.734481.submit parsl.executors.status_handling INFO: Allocated block ID 2362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2362 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2362 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2362.1709793922.734917.submit parsl.executors.status_handling INFO: Allocated block ID 2363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2363 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2363 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2363.1709793922.745305.submit parsl.executors.status_handling INFO: Allocated block ID 2364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2364 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2364 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2364.1709793922.7457607.submit parsl.executors.status_handling INFO: Allocated block ID 2365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2365 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2365 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2365.1709793922.7461982.submit parsl.executors.status_handling INFO: Allocated block ID 2366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2366 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2366 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2366.1709793922.7466333.submit parsl.executors.status_handling INFO: Allocated block ID 2367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2367 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2367 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2367.1709793922.7470682.submit parsl.executors.status_handling INFO: Allocated block ID 2368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2368 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2368 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2368.1709793922.7474928.submit parsl.executors.status_handling INFO: Allocated block ID 2369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2369 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2369 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2369.1709793922.7479239.submit parsl.executors.status_handling INFO: Allocated block ID 2370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2370 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2370 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2370.1709793922.748355.submit parsl.executors.status_handling INFO: Allocated block ID 2371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2371 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2371 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2371.1709793922.7487755.submit parsl.executors.status_handling INFO: Allocated block ID 2372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2372 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2372 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2372.1709793922.749209.submit parsl.executors.status_handling INFO: Allocated block ID 2373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2373 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2373 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2373.1709793922.7496305.submit parsl.executors.status_handling INFO: Allocated block ID 2374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2374 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2374 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2374.1709793922.7500587.submit parsl.executors.status_handling INFO: Allocated block ID 2375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2375 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2375 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2375.1709793922.7504816.submit parsl.executors.status_handling INFO: Allocated block ID 2376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2376 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2376 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2376.1709793922.7509093.submit parsl.executors.status_handling INFO: Allocated block ID 2377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2377 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2377 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2377.1709793922.751332.submit parsl.executors.status_handling INFO: Allocated block ID 2378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2378 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2378 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2378.1709793922.7517564.submit parsl.executors.status_handling INFO: Allocated block ID 2379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2379 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2379 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2379.1709793922.752185.submit parsl.executors.status_handling INFO: Allocated block ID 2380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2380 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2380 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2380.1709793922.7526016.submit parsl.executors.status_handling INFO: Allocated block ID 2381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2381 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2381 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2381.1709793922.753028.submit parsl.executors.status_handling INFO: Allocated block ID 2382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2382 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2382 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2382.1709793922.7534573.submit parsl.executors.status_handling INFO: Allocated block ID 2383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2383 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2383 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2383.1709793922.753888.submit parsl.executors.status_handling INFO: Allocated block ID 2384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2384 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2384 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2384.1709793922.7543173.submit parsl.executors.status_handling INFO: Allocated block ID 2385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2385 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2385 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2385.1709793922.7547398.submit parsl.executors.status_handling INFO: Allocated block ID 2386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2386 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2386 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2386.1709793922.7642367.submit parsl.executors.status_handling INFO: Allocated block ID 2387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2387 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2387 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2387.1709793922.76469.submit parsl.executors.status_handling INFO: Allocated block ID 2388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2388 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2388 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2388.1709793922.765167.submit parsl.executors.status_handling INFO: Allocated block ID 2389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2389 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2389 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2389.1709793922.7656035.submit parsl.executors.status_handling INFO: Allocated block ID 2390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2390 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2390 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2390.1709793922.7660358.submit parsl.executors.status_handling INFO: Allocated block ID 2391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2391 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2391 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2391.1709793922.7664616.submit parsl.executors.status_handling INFO: Allocated block ID 2392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2392 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2392 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2392.1709793922.766891.submit parsl.executors.status_handling INFO: Allocated block ID 2393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2393 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2393 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2393.1709793922.7673142.submit parsl.executors.status_handling INFO: Allocated block ID 2394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2394 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2394 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2394.1709793922.7677424.submit parsl.executors.status_handling INFO: Allocated block ID 2395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2395 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2395 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2395.1709793922.768175.submit parsl.executors.status_handling INFO: Allocated block ID 2396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2396 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2396 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2396.1709793922.7686024.submit parsl.executors.status_handling INFO: Allocated block ID 2397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2397 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2397 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2397.1709793922.7690363.submit parsl.executors.status_handling INFO: Allocated block ID 2398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2398 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2398 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2398.1709793922.7694592.submit parsl.executors.status_handling INFO: Allocated block ID 2399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2399 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2399 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2399.1709793922.7698832.submit parsl.executors.status_handling INFO: Allocated block ID 2400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2400 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2400 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2400.1709793922.7703016.submit parsl.executors.status_handling INFO: Allocated block ID 2401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2401 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2401 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2401.1709793922.770721.submit parsl.executors.status_handling INFO: Allocated block ID 2402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2402 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2402 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2402.1709793922.7711508.submit parsl.executors.status_handling INFO: Allocated block ID 2403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2403 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2403 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2403.1709793922.7715693.submit parsl.executors.status_handling INFO: Allocated block ID 2404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2404 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2404 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2404.1709793922.7719922.submit parsl.executors.status_handling INFO: Allocated block ID 2405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2405 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2405 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2405.1709793922.772412.submit parsl.executors.status_handling INFO: Allocated block ID 2406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2406 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2406 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2406.1709793922.7728431.submit parsl.executors.status_handling INFO: Allocated block ID 2407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2407 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2407 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2407.1709793922.7732668.submit parsl.executors.status_handling INFO: Allocated block ID 2408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2408 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2408 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2408.1709793922.7736938.submit parsl.executors.status_handling INFO: Allocated block ID 2409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2409 2024-03-06 22:45:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2409 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2409.1709793922.7830899.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faa762e88f43d4b47c73f7fb65f98366 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/00ff130c-13a9-46ba-9413-bdd64bb7b20d_biasIsr_1123734_47.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123734/00ff130c-13a9-46ba-9413-bdd64bb7b20d_biasIsr_1123734_47.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4edde5e12105d20e0feabcdd4325dc9 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/d5addc9c-3afd-48d9-b7e9-5bc6264f9151_biasIsr_1123738_47.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/d5addc9c-3afd-48d9-b7e9-5bc6264f9151_biasIsr_1123738_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1352 parsl.dataflow.strategy DEBUG: Executor multi has 1352 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2410 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2410 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2410.1709793927.6445842.submit parsl.executors.status_handling INFO: Allocated block ID 2411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2411 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2411 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2411.1709793927.6450722.submit parsl.executors.status_handling INFO: Allocated block ID 2412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2412 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2412 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2412.1709793927.6455145.submit parsl.executors.status_handling INFO: Allocated block ID 2413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2413 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2413 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2413.1709793927.6459694.submit parsl.executors.status_handling INFO: Allocated block ID 2414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2414 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2414 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2414.1709793927.6464186.submit parsl.executors.status_handling INFO: Allocated block ID 2415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2415 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2415 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2415.1709793927.6468632.submit parsl.executors.status_handling INFO: Allocated block ID 2416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2416 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2416 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2416.1709793927.647308.submit parsl.executors.status_handling INFO: Allocated block ID 2417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2417 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2417 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2417.1709793927.6477325.submit parsl.executors.status_handling INFO: Allocated block ID 2418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2418 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2418 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2418.1709793927.6481633.submit parsl.executors.status_handling INFO: Allocated block ID 2419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2419 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2419 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2419.1709793927.6485922.submit parsl.executors.status_handling INFO: Allocated block ID 2420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2420 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2420 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2420.1709793927.6490185.submit parsl.executors.status_handling INFO: Allocated block ID 2421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2421 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2421 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2421.1709793927.649436.submit parsl.executors.status_handling INFO: Allocated block ID 2422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2422 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2422 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2422.1709793927.6498656.submit parsl.executors.status_handling INFO: Allocated block ID 2423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2423 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2423 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2423.1709793927.6502879.submit parsl.executors.status_handling INFO: Allocated block ID 2424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2424 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2424 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2424.1709793927.6507049.submit parsl.executors.status_handling INFO: Allocated block ID 2425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2425 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2425 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2425.1709793927.6511393.submit parsl.executors.status_handling INFO: Allocated block ID 2426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2426 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2426 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2426.1709793927.6515648.submit parsl.executors.status_handling INFO: Allocated block ID 2427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2427 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2427 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2427.1709793927.6519978.submit parsl.executors.status_handling INFO: Allocated block ID 2428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2428 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2428 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2428.1709793927.6524243.submit parsl.executors.status_handling INFO: Allocated block ID 2429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2429 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2429 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2429.1709793927.6528602.submit parsl.executors.status_handling INFO: Allocated block ID 2430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2430 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2430 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2430.1709793927.6533108.submit parsl.executors.status_handling INFO: Allocated block ID 2431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2431 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2431 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2431.1709793927.6537445.submit parsl.executors.status_handling INFO: Allocated block ID 2432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2432 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2432 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2432.1709793927.6541893.submit parsl.executors.status_handling INFO: Allocated block ID 2433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2433 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2433 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2433.1709793927.6546233.submit parsl.executors.status_handling INFO: Allocated block ID 2434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2434 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2434 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2434.1709793927.6550808.submit parsl.executors.status_handling INFO: Allocated block ID 2435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2435 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2435 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2435.1709793927.6555161.submit parsl.executors.status_handling INFO: Allocated block ID 2436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2436 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2436 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2436.1709793927.6559541.submit parsl.executors.status_handling INFO: Allocated block ID 2437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2437 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2437 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2437.1709793927.6563954.submit parsl.executors.status_handling INFO: Allocated block ID 2438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2438 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2438 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2438.1709793927.6698627.submit parsl.executors.status_handling INFO: Allocated block ID 2439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2439 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2439 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2439.1709793927.670378.submit parsl.executors.status_handling INFO: Allocated block ID 2440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2440 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2440 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2440.1709793927.6708252.submit parsl.executors.status_handling INFO: Allocated block ID 2441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2441 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2441 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2441.1709793927.6712582.submit parsl.executors.status_handling INFO: Allocated block ID 2442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2442 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2442 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2442.1709793927.6716816.submit parsl.executors.status_handling INFO: Allocated block ID 2443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2443 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2443 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2443.1709793927.6721156.submit parsl.executors.status_handling INFO: Allocated block ID 2444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2444 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2444 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2444.1709793927.6725392.submit parsl.executors.status_handling INFO: Allocated block ID 2445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2445 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2445 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2445.1709793927.672969.submit parsl.executors.status_handling INFO: Allocated block ID 2446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2446 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2446 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2446.1709793927.673394.submit parsl.executors.status_handling INFO: Allocated block ID 2447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2447 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2447 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2447.1709793927.6738222.submit parsl.executors.status_handling INFO: Allocated block ID 2448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2448 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2448 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2448.1709793927.6742454.submit parsl.executors.status_handling INFO: Allocated block ID 2449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2449 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2449 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2449.1709793927.6746736.submit parsl.executors.status_handling INFO: Allocated block ID 2450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2450 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2450 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2450.1709793927.6751037.submit parsl.executors.status_handling INFO: Allocated block ID 2451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2451 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2451 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2451.1709793927.675535.submit parsl.executors.status_handling INFO: Allocated block ID 2452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2452 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2452 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2452.1709793927.675966.submit parsl.executors.status_handling INFO: Allocated block ID 2453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2453 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2453 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2453.1709793927.6763875.submit parsl.executors.status_handling INFO: Allocated block ID 2454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2454 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2454 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2454.1709793927.6768126.submit parsl.executors.status_handling INFO: Allocated block ID 2455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2455 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2455 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2455.1709793927.6772337.submit parsl.executors.status_handling INFO: Allocated block ID 2456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2456 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2456 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2456.1709793927.6776536.submit parsl.executors.status_handling INFO: Allocated block ID 2457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2457 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2457 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2457.1709793927.6780875.submit parsl.executors.status_handling INFO: Allocated block ID 2458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2458 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2458 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2458.1709793927.6785126.submit parsl.executors.status_handling INFO: Allocated block ID 2459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2459 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2459 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2459.1709793927.6789372.submit parsl.executors.status_handling INFO: Allocated block ID 2460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2460 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2460 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2460.1709793927.6793568.submit parsl.executors.status_handling INFO: Allocated block ID 2461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2461 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2461 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2461.1709793927.6797853.submit parsl.executors.status_handling INFO: Allocated block ID 2462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2462 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2462 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2462.1709793927.680236.submit parsl.executors.status_handling INFO: Allocated block ID 2463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2463 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2463 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2463.1709793927.6806757.submit parsl.executors.status_handling INFO: Allocated block ID 2464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2464 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2464 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2464.1709793927.681127.submit parsl.executors.status_handling INFO: Allocated block ID 2465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2465 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2465 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2465.1709793927.681553.submit parsl.executors.status_handling INFO: Allocated block ID 2466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2466 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2466 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2466.1709793927.6819892.submit parsl.executors.status_handling INFO: Allocated block ID 2467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2467 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2467 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2467.1709793927.682415.submit parsl.executors.status_handling INFO: Allocated block ID 2468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2468 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2468 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2468.1709793927.695901.submit parsl.executors.status_handling INFO: Allocated block ID 2469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2469 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2469 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2469.1709793927.696381.submit parsl.executors.status_handling INFO: Allocated block ID 2470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2470 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2470 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2470.1709793927.696823.submit parsl.executors.status_handling INFO: Allocated block ID 2471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2471 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2471 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2471.1709793927.6972516.submit parsl.executors.status_handling INFO: Allocated block ID 2472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2472 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2472 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2472.1709793927.6976895.submit parsl.executors.status_handling INFO: Allocated block ID 2473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2473 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2473 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2473.1709793927.698137.submit parsl.executors.status_handling INFO: Allocated block ID 2474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2474 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2474 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2474.1709793927.6985667.submit parsl.executors.status_handling INFO: Allocated block ID 2475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2475 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2475 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2475.1709793927.6990001.submit parsl.executors.status_handling INFO: Allocated block ID 2476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2476 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2476 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2476.1709793927.6994298.submit parsl.executors.status_handling INFO: Allocated block ID 2477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2477 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2477 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2477.1709793927.6998594.submit parsl.executors.status_handling INFO: Allocated block ID 2478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2478 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2478 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2478.1709793927.7002819.submit parsl.executors.status_handling INFO: Allocated block ID 2479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2479 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2479 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2479.1709793927.7007053.submit parsl.executors.status_handling INFO: Allocated block ID 2480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2480 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2480 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2480.1709793927.701134.submit parsl.executors.status_handling INFO: Allocated block ID 2481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2481 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2481 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2481.1709793927.7015564.submit parsl.executors.status_handling INFO: Allocated block ID 2482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2482 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2482 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2482.1709793927.7019897.submit parsl.executors.status_handling INFO: Allocated block ID 2483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2483 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2483 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2483.1709793927.7024329.submit parsl.executors.status_handling INFO: Allocated block ID 2484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2484 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2484 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2484.1709793927.702876.submit parsl.executors.status_handling INFO: Allocated block ID 2485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2485 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2485 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2485.1709793927.7033186.submit parsl.executors.status_handling INFO: Allocated block ID 2486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2486 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2486 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2486.1709793927.703767.submit parsl.executors.status_handling INFO: Allocated block ID 2487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2487 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2487 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2487.1709793927.704231.submit parsl.executors.status_handling INFO: Allocated block ID 2488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2488 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2488 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2488.1709793927.7046757.submit parsl.executors.status_handling INFO: Allocated block ID 2489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2489 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2489 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2489.1709793927.705106.submit parsl.executors.status_handling INFO: Allocated block ID 2490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2490 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2490 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2490.1709793927.705531.submit parsl.executors.status_handling INFO: Allocated block ID 2491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2491 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2491 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2491.1709793927.7059667.submit parsl.executors.status_handling INFO: Allocated block ID 2492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2492 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2492 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2492.1709793927.706396.submit parsl.executors.status_handling INFO: Allocated block ID 2493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2493 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2493 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2493.1709793927.706833.submit parsl.executors.status_handling INFO: Allocated block ID 2494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2494 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2494 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2494.1709793927.7072659.submit parsl.executors.status_handling INFO: Allocated block ID 2495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2495 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2495 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2495.1709793927.7076933.submit parsl.executors.status_handling INFO: Allocated block ID 2496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2496 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2496 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2496.1709793927.7081287.submit parsl.executors.status_handling INFO: Allocated block ID 2497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2497 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2497 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2497.1709793927.7085655.submit parsl.executors.status_handling INFO: Allocated block ID 2498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2498 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2498 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2498.1709793927.7220688.submit parsl.executors.status_handling INFO: Allocated block ID 2499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2499 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2499 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2499.1709793927.7225332.submit parsl.executors.status_handling INFO: Allocated block ID 2500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2500 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2500 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2500.1709793927.7229831.submit parsl.executors.status_handling INFO: Allocated block ID 2501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2501 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2501 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2501.1709793927.7234173.submit parsl.executors.status_handling INFO: Allocated block ID 2502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2502 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2502 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2502.1709793927.723857.submit parsl.executors.status_handling INFO: Allocated block ID 2503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2503 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2503 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2503.1709793927.7242913.submit parsl.executors.status_handling INFO: Allocated block ID 2504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2504 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2504 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2504.1709793927.7247195.submit parsl.executors.status_handling INFO: Allocated block ID 2505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2505 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2505 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2505.1709793927.7251594.submit parsl.executors.status_handling INFO: Allocated block ID 2506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2506 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2506 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2506.1709793927.7255957.submit parsl.executors.status_handling INFO: Allocated block ID 2507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2507 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2507 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2507.1709793927.7260382.submit parsl.executors.status_handling INFO: Allocated block ID 2508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2508 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2508 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2508.1709793927.7264688.submit parsl.executors.status_handling INFO: Allocated block ID 2509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2509 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2509 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2509.1709793927.7269087.submit parsl.executors.status_handling INFO: Allocated block ID 2510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2510 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2510 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2510.1709793927.7273407.submit parsl.executors.status_handling INFO: Allocated block ID 2511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2511 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2511 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2511.1709793927.7277725.submit parsl.executors.status_handling INFO: Allocated block ID 2512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2512 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2512 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2512.1709793927.7282128.submit parsl.executors.status_handling INFO: Allocated block ID 2513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2513 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2513 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2513.1709793927.728642.submit parsl.executors.status_handling INFO: Allocated block ID 2514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2514 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2514 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2514.1709793927.7290828.submit parsl.executors.status_handling INFO: Allocated block ID 2515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2515 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2515 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2515.1709793927.7295115.submit parsl.executors.status_handling INFO: Allocated block ID 2516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2516 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2516 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2516.1709793927.7299445.submit parsl.executors.status_handling INFO: Allocated block ID 2517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2517 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2517 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2517.1709793927.7303789.submit parsl.executors.status_handling INFO: Allocated block ID 2518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2518 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2518 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2518.1709793927.730814.submit parsl.executors.status_handling INFO: Allocated block ID 2519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2519 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2519 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2519.1709793927.7312434.submit parsl.executors.status_handling INFO: Allocated block ID 2520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2520 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2520 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2520.1709793927.7316692.submit parsl.executors.status_handling INFO: Allocated block ID 2521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2521 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2521 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2521.1709793927.7321033.submit parsl.executors.status_handling INFO: Allocated block ID 2522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2522 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2522 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2522.1709793927.732537.submit parsl.executors.status_handling INFO: Allocated block ID 2523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2523 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2523 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2523.1709793927.7329693.submit parsl.executors.status_handling INFO: Allocated block ID 2524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2524 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2524 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2524.1709793927.7334003.submit parsl.executors.status_handling INFO: Allocated block ID 2525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2525 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2525 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2525.1709793927.733834.submit parsl.executors.status_handling INFO: Allocated block ID 2526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2526 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2526 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2526.1709793927.7342618.submit parsl.executors.status_handling INFO: Allocated block ID 2527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2527 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2527 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2527.1709793927.734687.submit parsl.executors.status_handling INFO: Allocated block ID 2528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2528 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2528 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2528.1709793927.7481987.submit parsl.executors.status_handling INFO: Allocated block ID 2529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2529 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2529 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2529.1709793927.7486403.submit parsl.executors.status_handling INFO: Allocated block ID 2530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2530 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2530 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2530.1709793927.7490842.submit parsl.executors.status_handling INFO: Allocated block ID 2531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2531 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2531 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2531.1709793927.7495148.submit parsl.executors.status_handling INFO: Allocated block ID 2532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2532 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2532 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2532.1709793927.749965.submit parsl.executors.status_handling INFO: Allocated block ID 2533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2533 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2533 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2533.1709793927.7504091.submit parsl.executors.status_handling INFO: Allocated block ID 2534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2534 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2534 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2534.1709793927.7508516.submit parsl.executors.status_handling INFO: Allocated block ID 2535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2535 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2535 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2535.1709793927.7512867.submit parsl.executors.status_handling INFO: Allocated block ID 2536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2536 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2536 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2536.1709793927.7517257.submit parsl.executors.status_handling INFO: Allocated block ID 2537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2537 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2537 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2537.1709793927.7521608.submit parsl.executors.status_handling INFO: Allocated block ID 2538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2538 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2538 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2538.1709793927.752587.submit parsl.executors.status_handling INFO: Allocated block ID 2539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2539 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2539 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2539.1709793927.7530262.submit parsl.executors.status_handling INFO: Allocated block ID 2540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2540 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2540 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2540.1709793927.7534516.submit parsl.executors.status_handling INFO: Allocated block ID 2541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2541 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2541 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2541.1709793927.753887.submit parsl.executors.status_handling INFO: Allocated block ID 2542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2542 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2542 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2542.1709793927.7543194.submit parsl.executors.status_handling INFO: Allocated block ID 2543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2543 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2543 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2543.1709793927.7547462.submit parsl.executors.status_handling INFO: Allocated block ID 2544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2544 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2544 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2544.1709793927.755182.submit parsl.executors.status_handling INFO: Allocated block ID 2545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2545 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2545 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2545.1709793927.755611.submit parsl.executors.status_handling INFO: Allocated block ID 2546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2546 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2546 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2546.1709793927.756042.submit parsl.executors.status_handling INFO: Allocated block ID 2547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2547 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2547 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2547.1709793927.7564714.submit parsl.executors.status_handling INFO: Allocated block ID 2548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2548 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2548 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2548.1709793927.7569063.submit parsl.executors.status_handling INFO: Allocated block ID 2549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2549 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2549 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2549.1709793927.75733.submit parsl.executors.status_handling INFO: Allocated block ID 2550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2550 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2550 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2550.1709793927.7577484.submit parsl.executors.status_handling INFO: Allocated block ID 2551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2551 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2551 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2551.1709793927.7581809.submit parsl.executors.status_handling INFO: Allocated block ID 2552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2552 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2552 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2552.1709793927.7586021.submit parsl.executors.status_handling INFO: Allocated block ID 2553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2553 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2553 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2553.1709793927.759027.submit parsl.executors.status_handling INFO: Allocated block ID 2554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2554 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2554 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2554.1709793927.7594535.submit parsl.executors.status_handling INFO: Allocated block ID 2555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2555 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2555 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2555.1709793927.7598798.submit parsl.executors.status_handling INFO: Allocated block ID 2556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2556 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2556 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2556.1709793927.7602997.submit parsl.executors.status_handling INFO: Allocated block ID 2557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2557 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2557 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2557.1709793927.7607327.submit parsl.executors.status_handling INFO: Allocated block ID 2558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2558 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2558 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2558.1709793927.7742693.submit parsl.executors.status_handling INFO: Allocated block ID 2559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2559 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2559 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2559.1709793927.7747169.submit parsl.executors.status_handling INFO: Allocated block ID 2560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2560 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2560 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2560.1709793927.7751849.submit parsl.executors.status_handling INFO: Allocated block ID 2561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2561 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2561 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2561.1709793927.7756245.submit parsl.executors.status_handling INFO: Allocated block ID 2562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2562 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2562 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2562.1709793927.7760751.submit parsl.executors.status_handling INFO: Allocated block ID 2563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2563 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2563 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2563.1709793927.7765028.submit parsl.executors.status_handling INFO: Allocated block ID 2564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2564 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2564 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2564.1709793927.7769322.submit parsl.executors.status_handling INFO: Allocated block ID 2565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2565 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2565 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2565.1709793927.7773554.submit parsl.executors.status_handling INFO: Allocated block ID 2566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2566 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2566 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2566.1709793927.777778.submit parsl.executors.status_handling INFO: Allocated block ID 2567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2567 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2567 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2567.1709793927.7782097.submit parsl.executors.status_handling INFO: Allocated block ID 2568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2568 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2568 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2568.1709793927.7786314.submit parsl.executors.status_handling INFO: Allocated block ID 2569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2569 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2569 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2569.1709793927.779065.submit parsl.executors.status_handling INFO: Allocated block ID 2570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2570 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2570 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2570.1709793927.779493.submit parsl.executors.status_handling INFO: Allocated block ID 2571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2571 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2571 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2571.1709793927.7799213.submit parsl.executors.status_handling INFO: Allocated block ID 2572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2572 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2572 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2572.1709793927.7803543.submit parsl.executors.status_handling INFO: Allocated block ID 2573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2573 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2573 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2573.1709793927.7807798.submit parsl.executors.status_handling INFO: Allocated block ID 2574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2574 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2574 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2574.1709793927.781207.submit parsl.executors.status_handling INFO: Allocated block ID 2575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2575 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2575 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2575.1709793927.781652.submit parsl.executors.status_handling INFO: Allocated block ID 2576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2576 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2576 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2576.1709793927.782101.submit parsl.executors.status_handling INFO: Allocated block ID 2577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2577 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2577 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2577.1709793927.782528.submit parsl.executors.status_handling INFO: Allocated block ID 2578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2578 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2578 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2578.1709793927.7829633.submit parsl.executors.status_handling INFO: Allocated block ID 2579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2579 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2579 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2579.1709793927.7833846.submit parsl.executors.status_handling INFO: Allocated block ID 2580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2580 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2580 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2580.1709793927.7838054.submit parsl.executors.status_handling INFO: Allocated block ID 2581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2581 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2581 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2581.1709793927.7842662.submit parsl.executors.status_handling INFO: Allocated block ID 2582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2582 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2582 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2582.1709793927.7847006.submit parsl.executors.status_handling INFO: Allocated block ID 2583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2583 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2583 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2583.1709793927.7851431.submit parsl.executors.status_handling INFO: Allocated block ID 2584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2584 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2584 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2584.1709793927.785584.submit parsl.executors.status_handling INFO: Allocated block ID 2585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2585 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2585 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2585.1709793927.7860227.submit parsl.executors.status_handling INFO: Allocated block ID 2586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2586 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2586 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2586.1709793927.7864666.submit parsl.executors.status_handling INFO: Allocated block ID 2587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2587 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2587 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2587.1709793927.799963.submit parsl.executors.status_handling INFO: Allocated block ID 2588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2588 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2588 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2588.1709793927.800427.submit parsl.executors.status_handling INFO: Allocated block ID 2589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2589 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2589 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2589.1709793927.8008657.submit parsl.executors.status_handling INFO: Allocated block ID 2590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2590 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2590 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2590.1709793927.8012946.submit parsl.executors.status_handling INFO: Allocated block ID 2591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2591 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2591 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2591.1709793927.8017173.submit parsl.executors.status_handling INFO: Allocated block ID 2592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2592 2024-03-06 22:45:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2592 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2592.1709793927.8021545.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1352 parsl.dataflow.strategy DEBUG: Executor multi has 1352 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2593 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2593 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2593.1709793932.650054.submit parsl.executors.status_handling INFO: Allocated block ID 2594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2594 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2594 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2594.1709793932.6521666.submit parsl.executors.status_handling INFO: Allocated block ID 2595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2595 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2595 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2595.1709793932.6534448.submit parsl.executors.status_handling INFO: Allocated block ID 2596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2596 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2596 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2596.1709793932.6545064.submit parsl.executors.status_handling INFO: Allocated block ID 2597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2597 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2597 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2597.1709793932.6554952.submit parsl.executors.status_handling INFO: Allocated block ID 2598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2598 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2598 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2598.1709793932.6564777.submit parsl.executors.status_handling INFO: Allocated block ID 2599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2599 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2599 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2599.1709793932.657475.submit parsl.executors.status_handling INFO: Allocated block ID 2600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2600 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2600 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2600.1709793932.658453.submit parsl.executors.status_handling INFO: Allocated block ID 2601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2601 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2601 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2601.1709793932.6594322.submit parsl.executors.status_handling INFO: Allocated block ID 2602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2602 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2602 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2602.1709793932.6604204.submit parsl.executors.status_handling INFO: Allocated block ID 2603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2603 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2603 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2603.1709793932.6614091.submit parsl.executors.status_handling INFO: Allocated block ID 2604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2604 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2604 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2604.1709793932.6624045.submit parsl.executors.status_handling INFO: Allocated block ID 2605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2605 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2605 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2605.1709793932.6629999.submit parsl.executors.status_handling INFO: Allocated block ID 2606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2606 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2606 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2606.1709793932.6634471.submit parsl.executors.status_handling INFO: Allocated block ID 2607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2607 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2607 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2607.1709793932.6638994.submit parsl.executors.status_handling INFO: Allocated block ID 2608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2608 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2608 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2608.1709793932.6643403.submit parsl.executors.status_handling INFO: Allocated block ID 2609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2609 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2609 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2609.1709793932.6647584.submit parsl.executors.status_handling INFO: Allocated block ID 2610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2610 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2610 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2610.1709793932.6651666.submit parsl.executors.status_handling INFO: Allocated block ID 2611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2611 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2611 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2611.1709793932.6655705.submit parsl.executors.status_handling INFO: Allocated block ID 2612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2612 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2612 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2612.1709793932.666004.submit parsl.executors.status_handling INFO: Allocated block ID 2613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2613 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2613 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2613.1709793932.6664293.submit parsl.executors.status_handling INFO: Allocated block ID 2614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2614 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2614 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2614.1709793932.6668584.submit parsl.executors.status_handling INFO: Allocated block ID 2615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2615 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2615 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2615.1709793932.6672883.submit parsl.executors.status_handling INFO: Allocated block ID 2616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2616 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2616 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2616.1709793932.6676953.submit parsl.executors.status_handling INFO: Allocated block ID 2617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2617 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2617 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2617.1709793932.6681406.submit parsl.executors.status_handling INFO: Allocated block ID 2618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2618 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2618 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2618.1709793932.668593.submit parsl.executors.status_handling INFO: Allocated block ID 2619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2619 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2619 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2619.1709793932.6690454.submit parsl.executors.status_handling INFO: Allocated block ID 2620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2620 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2620 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2620.1709793932.669492.submit parsl.executors.status_handling INFO: Allocated block ID 2621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2621 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2621 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2621.1709793932.66994.submit parsl.executors.status_handling INFO: Allocated block ID 2622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2622 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2622 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2622.1709793932.670384.submit parsl.executors.status_handling INFO: Allocated block ID 2623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2623 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2623 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2623.1709793932.6708179.submit parsl.executors.status_handling INFO: Allocated block ID 2624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2624 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2624 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2624.1709793932.67127.submit parsl.executors.status_handling INFO: Allocated block ID 2625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2625 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2625 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2625.1709793932.6717217.submit parsl.executors.status_handling INFO: Allocated block ID 2626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2626 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2626 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2626.1709793932.672194.submit parsl.executors.status_handling INFO: Allocated block ID 2627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2627 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2627 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2627.1709793932.6726391.submit parsl.executors.status_handling INFO: Allocated block ID 2628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2628 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2628 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2628.1709793932.6730788.submit parsl.executors.status_handling INFO: Allocated block ID 2629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2629 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2629 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2629.1709793932.6734984.submit parsl.executors.status_handling INFO: Allocated block ID 2630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2630 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2630 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2630.1709793932.67393.submit parsl.executors.status_handling INFO: Allocated block ID 2631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2631 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2631 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2631.1709793932.6743531.submit parsl.executors.status_handling INFO: Allocated block ID 2632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2632 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2632 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2632.1709793932.6747787.submit parsl.executors.status_handling INFO: Allocated block ID 2633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2633 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2633 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2633.1709793932.6752121.submit parsl.executors.status_handling INFO: Allocated block ID 2634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2634 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2634 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2634.1709793932.6756132.submit parsl.executors.status_handling INFO: Allocated block ID 2635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2635 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2635 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2635.1709793932.6760225.submit parsl.executors.status_handling INFO: Allocated block ID 2636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2636 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2636 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2636.1709793932.676431.submit parsl.executors.status_handling INFO: Allocated block ID 2637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2637 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2637 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2637.1709793932.6768441.submit parsl.executors.status_handling INFO: Allocated block ID 2638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2638 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2638 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2638.1709793932.6772711.submit parsl.executors.status_handling INFO: Allocated block ID 2639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2639 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2639 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2639.1709793932.6776943.submit parsl.executors.status_handling INFO: Allocated block ID 2640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2640 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2640 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2640.1709793932.6781359.submit parsl.executors.status_handling INFO: Allocated block ID 2641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2641 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2641 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2641.1709793932.6785784.submit parsl.executors.status_handling INFO: Allocated block ID 2642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2642 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2642 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2642.1709793932.6790192.submit parsl.executors.status_handling INFO: Allocated block ID 2643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2643 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2643 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2643.1709793932.679451.submit parsl.executors.status_handling INFO: Allocated block ID 2644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2644 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2644 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2644.1709793932.679905.submit parsl.executors.status_handling INFO: Allocated block ID 2645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2645 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2645 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2645.1709793932.680347.submit parsl.executors.status_handling INFO: Allocated block ID 2646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2646 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2646 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2646.1709793932.6807852.submit parsl.executors.status_handling INFO: Allocated block ID 2647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2647 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2647 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2647.1709793932.6812308.submit parsl.executors.status_handling INFO: Allocated block ID 2648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2648 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2648 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2648.1709793932.6816912.submit parsl.executors.status_handling INFO: Allocated block ID 2649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2649 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2649 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2649.1709793932.6821322.submit parsl.executors.status_handling INFO: Allocated block ID 2650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2650 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2650 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2650.1709793932.6825619.submit parsl.executors.status_handling INFO: Allocated block ID 2651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2651 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2651 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2651.1709793932.683003.submit parsl.executors.status_handling INFO: Allocated block ID 2652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2652 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2652 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2652.1709793932.6834512.submit parsl.executors.status_handling INFO: Allocated block ID 2653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2653 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2653 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2653.1709793932.6838858.submit parsl.executors.status_handling INFO: Allocated block ID 2654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2654 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2654 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2654.1709793932.6843174.submit parsl.executors.status_handling INFO: Allocated block ID 2655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2655 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2655 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2655.1709793932.6847446.submit parsl.executors.status_handling INFO: Allocated block ID 2656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2656 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2656 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2656.1709793932.6851752.submit parsl.executors.status_handling INFO: Allocated block ID 2657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2657 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2657 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2657.1709793932.6856012.submit parsl.executors.status_handling INFO: Allocated block ID 2658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2658 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2658 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2658.1709793932.6860375.submit parsl.executors.status_handling INFO: Allocated block ID 2659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2659 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2659 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2659.1709793932.68644.submit parsl.executors.status_handling INFO: Allocated block ID 2660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2660 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2660 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2660.1709793932.6868474.submit parsl.executors.status_handling INFO: Allocated block ID 2661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2661 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2661 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2661.1709793932.6872509.submit parsl.executors.status_handling INFO: Allocated block ID 2662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2662 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2662 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2662.1709793932.6876502.submit parsl.executors.status_handling INFO: Allocated block ID 2663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2663 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2663 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2663.1709793932.6881065.submit parsl.executors.status_handling INFO: Allocated block ID 2664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2664 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2664 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2664.1709793932.6885452.submit parsl.executors.status_handling INFO: Allocated block ID 2665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2665 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2665 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2665.1709793932.6890018.submit parsl.executors.status_handling INFO: Allocated block ID 2666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2666 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2666 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2666.1709793932.689447.submit parsl.executors.status_handling INFO: Allocated block ID 2667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2667 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2667 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2667.1709793932.6898992.submit parsl.executors.status_handling INFO: Allocated block ID 2668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2668 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2668 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2668.1709793932.6903498.submit parsl.executors.status_handling INFO: Allocated block ID 2669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2669 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2669 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2669.1709793932.6907902.submit parsl.executors.status_handling INFO: Allocated block ID 2670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2670 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2670 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2670.1709793932.6912324.submit parsl.executors.status_handling INFO: Allocated block ID 2671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2671 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2671 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2671.1709793932.691727.submit parsl.executors.status_handling INFO: Allocated block ID 2672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2672 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2672 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2672.1709793932.6922054.submit parsl.executors.status_handling INFO: Allocated block ID 2673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2673 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2673 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2673.1709793932.6926632.submit parsl.executors.status_handling INFO: Allocated block ID 2674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2674 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2674 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2674.1709793932.693104.submit parsl.executors.status_handling INFO: Allocated block ID 2675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2675 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2675 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2675.1709793932.693534.submit parsl.executors.status_handling INFO: Allocated block ID 2676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2676 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2676 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2676.1709793932.6939783.submit parsl.executors.status_handling INFO: Allocated block ID 2677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2677 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2677 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2677.1709793932.694413.submit parsl.executors.status_handling INFO: Allocated block ID 2678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2678 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2678 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2678.1709793932.6948414.submit parsl.executors.status_handling INFO: Allocated block ID 2679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2679 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2679 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2679.1709793932.6952639.submit parsl.executors.status_handling INFO: Allocated block ID 2680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2680 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2680 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2680.1709793932.6957057.submit parsl.executors.status_handling INFO: Allocated block ID 2681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2681 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2681 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2681.1709793932.6961465.submit parsl.executors.status_handling INFO: Allocated block ID 2682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2682 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2682 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2682.1709793932.6966035.submit parsl.executors.status_handling INFO: Allocated block ID 2683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2683 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2683 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2683.1709793932.6970468.submit parsl.executors.status_handling INFO: Allocated block ID 2684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2684 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2684 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2684.1709793932.6974945.submit parsl.executors.status_handling INFO: Allocated block ID 2685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2685 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2685 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2685.1709793932.6979249.submit parsl.executors.status_handling INFO: Allocated block ID 2686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2686 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2686 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2686.1709793932.6983488.submit parsl.executors.status_handling INFO: Allocated block ID 2687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2687 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2687 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2687.1709793932.698769.submit parsl.executors.status_handling INFO: Allocated block ID 2688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2688 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2688 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2688.1709793932.699221.submit parsl.executors.status_handling INFO: Allocated block ID 2689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2689 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2689 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2689.1709793932.6996572.submit parsl.executors.status_handling INFO: Allocated block ID 2690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2690 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2690 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2690.1709793932.7001078.submit parsl.executors.status_handling INFO: Allocated block ID 2691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2691 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2691 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2691.1709793932.7005506.submit parsl.executors.status_handling INFO: Allocated block ID 2692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2692 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2692 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2692.1709793932.7009802.submit parsl.executors.status_handling INFO: Allocated block ID 2693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2693 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2693 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2693.1709793932.7014043.submit parsl.executors.status_handling INFO: Allocated block ID 2694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2694 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2694 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2694.1709793932.7018316.submit parsl.executors.status_handling INFO: Allocated block ID 2695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2695 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2695 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2695.1709793932.7022562.submit parsl.executors.status_handling INFO: Allocated block ID 2696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2696 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2696 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2696.1709793932.7026792.submit parsl.executors.status_handling INFO: Allocated block ID 2697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2697 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2697 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2697.1709793932.7031124.submit parsl.executors.status_handling INFO: Allocated block ID 2698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2698 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2698 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2698.1709793932.7035332.submit parsl.executors.status_handling INFO: Allocated block ID 2699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2699 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2699 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2699.1709793932.7039626.submit parsl.executors.status_handling INFO: Allocated block ID 2700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2700 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2700 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2700.1709793932.704392.submit parsl.executors.status_handling INFO: Allocated block ID 2701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2701 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2701 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2701.1709793932.704824.submit parsl.executors.status_handling INFO: Allocated block ID 2702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2702 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2702 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2702.1709793932.7052677.submit parsl.executors.status_handling INFO: Allocated block ID 2703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2703 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2703 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2703.1709793932.705702.submit parsl.executors.status_handling INFO: Allocated block ID 2704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2704 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2704 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2704.1709793932.7061393.submit parsl.executors.status_handling INFO: Allocated block ID 2705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2705 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2705 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2705.1709793932.706571.submit parsl.executors.status_handling INFO: Allocated block ID 2706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2706 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2706 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2706.1709793932.7070138.submit parsl.executors.status_handling INFO: Allocated block ID 2707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2707 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2707 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2707.1709793932.7074485.submit parsl.executors.status_handling INFO: Allocated block ID 2708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2708 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2708 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2708.1709793932.7078838.submit parsl.executors.status_handling INFO: Allocated block ID 2709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2709 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2709 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2709.1709793932.7083056.submit parsl.executors.status_handling INFO: Allocated block ID 2710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2710 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2710 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2710.1709793932.7087252.submit parsl.executors.status_handling INFO: Allocated block ID 2711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2711 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2711 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2711.1709793932.709152.submit parsl.executors.status_handling INFO: Allocated block ID 2712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2712 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2712 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2712.1709793932.7095847.submit parsl.executors.status_handling INFO: Allocated block ID 2713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2713 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2713 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2713.1709793932.7100103.submit parsl.executors.status_handling INFO: Allocated block ID 2714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2714 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2714 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2714.1709793932.7104328.submit parsl.executors.status_handling INFO: Allocated block ID 2715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2715 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2715 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2715.1709793932.7108564.submit parsl.executors.status_handling INFO: Allocated block ID 2716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2716 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2716 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2716.1709793932.7112796.submit parsl.executors.status_handling INFO: Allocated block ID 2717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2717 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2717 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2717.1709793932.711706.submit parsl.executors.status_handling INFO: Allocated block ID 2718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2718 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2718 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2718.1709793932.7121208.submit parsl.executors.status_handling INFO: Allocated block ID 2719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2719 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2719 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2719.1709793932.7125192.submit parsl.executors.status_handling INFO: Allocated block ID 2720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2720 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2720 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2720.1709793932.712923.submit parsl.executors.status_handling INFO: Allocated block ID 2721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2721 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2721 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2721.1709793932.7133212.submit parsl.executors.status_handling INFO: Allocated block ID 2722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2722 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2722 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2722.1709793932.7137191.submit parsl.executors.status_handling INFO: Allocated block ID 2723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2723 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2723 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2723.1709793932.7141566.submit parsl.executors.status_handling INFO: Allocated block ID 2724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2724 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2724 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2724.1709793932.7145846.submit parsl.executors.status_handling INFO: Allocated block ID 2725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2725 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2725 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2725.1709793932.7150116.submit parsl.executors.status_handling INFO: Allocated block ID 2726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2726 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2726 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2726.1709793932.7154293.submit parsl.executors.status_handling INFO: Allocated block ID 2727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2727 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2727 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2727.1709793932.7158492.submit parsl.executors.status_handling INFO: Allocated block ID 2728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2728 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2728 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2728.1709793932.7162552.submit parsl.executors.status_handling INFO: Allocated block ID 2729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2729 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2729 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2729.1709793932.716656.submit parsl.executors.status_handling INFO: Allocated block ID 2730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2730 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2730 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2730.1709793932.7170625.submit parsl.executors.status_handling INFO: Allocated block ID 2731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2731 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2731 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2731.1709793932.7174854.submit parsl.executors.status_handling INFO: Allocated block ID 2732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2732 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2732 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2732.1709793932.718048.submit parsl.executors.status_handling INFO: Allocated block ID 2733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2733 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2733 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2733.1709793932.718491.submit parsl.executors.status_handling INFO: Allocated block ID 2734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2734 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2734 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2734.1709793932.7189207.submit parsl.executors.status_handling INFO: Allocated block ID 2735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2735 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2735 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2735.1709793932.7193418.submit parsl.executors.status_handling INFO: Allocated block ID 2736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2736 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2736 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2736.1709793932.7197728.submit parsl.executors.status_handling INFO: Allocated block ID 2737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2737 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2737 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2737.1709793932.7202115.submit parsl.executors.status_handling INFO: Allocated block ID 2738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2738 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2738 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2738.1709793932.7206666.submit parsl.executors.status_handling INFO: Allocated block ID 2739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2739 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2739 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2739.1709793932.7211082.submit parsl.executors.status_handling INFO: Allocated block ID 2740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2740 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2740 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2740.1709793932.7215438.submit parsl.executors.status_handling INFO: Allocated block ID 2741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2741 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2741 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2741.1709793932.721979.submit parsl.executors.status_handling INFO: Allocated block ID 2742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2742 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2742 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2742.1709793932.7224195.submit parsl.executors.status_handling INFO: Allocated block ID 2743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2743 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2743 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2743.1709793932.7228537.submit parsl.executors.status_handling INFO: Allocated block ID 2744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2744 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2744 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2744.1709793932.723282.submit parsl.executors.status_handling INFO: Allocated block ID 2745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2745 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2745 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2745.1709793932.7237072.submit parsl.executors.status_handling INFO: Allocated block ID 2746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2746 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2746 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2746.1709793932.7241366.submit parsl.executors.status_handling INFO: Allocated block ID 2747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2747 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2747 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2747.1709793932.7245803.submit parsl.executors.status_handling INFO: Allocated block ID 2748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2748 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2748 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2748.1709793932.7254117.submit parsl.executors.status_handling INFO: Allocated block ID 2749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2749 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2749 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2749.1709793932.725893.submit parsl.executors.status_handling INFO: Allocated block ID 2750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2750 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2750 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2750.1709793932.7263253.submit parsl.executors.status_handling INFO: Allocated block ID 2751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2751 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2751 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2751.1709793932.726751.submit parsl.executors.status_handling INFO: Allocated block ID 2752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2752 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2752 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2752.1709793932.7271833.submit parsl.executors.status_handling INFO: Allocated block ID 2753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2753 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2753 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2753.1709793932.7276082.submit parsl.executors.status_handling INFO: Allocated block ID 2754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2754 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2754 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2754.1709793932.728047.submit parsl.executors.status_handling INFO: Allocated block ID 2755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2755 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2755 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2755.1709793932.7284946.submit parsl.executors.status_handling INFO: Allocated block ID 2756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2756 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2756 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2756.1709793932.7289457.submit parsl.executors.status_handling INFO: Allocated block ID 2757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2757 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2757 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2757.1709793932.7293844.submit parsl.executors.status_handling INFO: Allocated block ID 2758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2758 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2758 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2758.1709793932.7298315.submit parsl.executors.status_handling INFO: Allocated block ID 2759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2759 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2759 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2759.1709793932.7302523.submit parsl.executors.status_handling INFO: Allocated block ID 2760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2760 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2760 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2760.1709793932.73068.submit parsl.executors.status_handling INFO: Allocated block ID 2761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2761 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2761 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2761.1709793932.731131.submit parsl.executors.status_handling INFO: Allocated block ID 2762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2762 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2762 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2762.1709793932.7315717.submit parsl.executors.status_handling INFO: Allocated block ID 2763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2763 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2763 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2763.1709793932.732019.submit parsl.executors.status_handling INFO: Allocated block ID 2764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2764 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2764 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2764.1709793932.7324536.submit parsl.executors.status_handling INFO: Allocated block ID 2765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2765 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2765 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2765.1709793932.7328954.submit parsl.executors.status_handling INFO: Allocated block ID 2766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2766 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2766 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2766.1709793932.7333152.submit parsl.executors.status_handling INFO: Allocated block ID 2767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2767 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2767 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2767.1709793932.7337153.submit parsl.executors.status_handling INFO: Allocated block ID 2768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2768 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2768 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2768.1709793932.734143.submit parsl.executors.status_handling INFO: Allocated block ID 2769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2769 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2769 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2769.1709793932.7345715.submit parsl.executors.status_handling INFO: Allocated block ID 2770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2770 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2770 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2770.1709793932.7350006.submit parsl.executors.status_handling INFO: Allocated block ID 2771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2771 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2771 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2771.1709793932.7354233.submit parsl.executors.status_handling INFO: Allocated block ID 2772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2772 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2772 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2772.1709793932.735864.submit parsl.executors.status_handling INFO: Allocated block ID 2773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2773 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2773 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2773.1709793932.736282.submit parsl.executors.status_handling INFO: Allocated block ID 2774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2774 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2774 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2774.1709793932.7366836.submit parsl.executors.status_handling INFO: Allocated block ID 2775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2775 2024-03-06 22:45:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2775 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2775.1709793932.73709.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b82f77e8f9fdcf52354b8d642ba0e032 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/86ab7a95-ed63-4261-9b88-80e0c57d8ce4_biasIsr_1123833_47.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/86ab7a95-ed63-4261-9b88-80e0c57d8ce4_biasIsr_1123833_47.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5717c69642806b5220ded9282c773c6 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/56c80a84-185d-4894-bec6-db8b129a16f1_biasIsr_1123836_47.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/56c80a84-185d-4894-bec6-db8b129a16f1_biasIsr_1123836_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1350 parsl.dataflow.strategy DEBUG: Executor multi has 1350 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2776 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2776 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2776.1709793937.6456144.submit parsl.executors.status_handling INFO: Allocated block ID 2777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2777 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2777 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2777.1709793937.6461196.submit parsl.executors.status_handling INFO: Allocated block ID 2778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2778 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2778 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2778.1709793937.6465652.submit parsl.executors.status_handling INFO: Allocated block ID 2779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2779 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2779 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2779.1709793937.6470175.submit parsl.executors.status_handling INFO: Allocated block ID 2780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2780 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2780 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2780.1709793937.647452.submit parsl.executors.status_handling INFO: Allocated block ID 2781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2781 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2781 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2781.1709793937.647894.submit parsl.executors.status_handling INFO: Allocated block ID 2782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2782 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2782 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2782.1709793937.6483319.submit parsl.executors.status_handling INFO: Allocated block ID 2783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2783 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2783 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2783.1709793937.64876.submit parsl.executors.status_handling INFO: Allocated block ID 2784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2784 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2784 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2784.1709793937.6491969.submit parsl.executors.status_handling INFO: Allocated block ID 2785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2785 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2785 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2785.1709793937.6496222.submit parsl.executors.status_handling INFO: Allocated block ID 2786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2786 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2786 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2786.1709793937.6500516.submit parsl.executors.status_handling INFO: Allocated block ID 2787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2787 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2787 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2787.1709793937.650492.submit parsl.executors.status_handling INFO: Allocated block ID 2788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2788 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2788 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2788.1709793937.6509333.submit parsl.executors.status_handling INFO: Allocated block ID 2789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2789 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2789 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2789.1709793937.6513577.submit parsl.executors.status_handling INFO: Allocated block ID 2790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2790 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2790 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2790.1709793937.651781.submit parsl.executors.status_handling INFO: Allocated block ID 2791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2791 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2791 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2791.1709793937.6522312.submit parsl.executors.status_handling INFO: Allocated block ID 2792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2792 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2792 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2792.1709793937.6526644.submit parsl.executors.status_handling INFO: Allocated block ID 2793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2793 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2793 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2793.1709793937.6530964.submit parsl.executors.status_handling INFO: Allocated block ID 2794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2794 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2794 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2794.1709793937.6535316.submit parsl.executors.status_handling INFO: Allocated block ID 2795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2795 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2795 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2795.1709793937.6539736.submit parsl.executors.status_handling INFO: Allocated block ID 2796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2796 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2796 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2796.1709793937.6544.submit parsl.executors.status_handling INFO: Allocated block ID 2797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2797 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2797 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2797.1709793937.6549363.submit parsl.executors.status_handling INFO: Allocated block ID 2798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2798 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2798 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2798.1709793937.6553695.submit parsl.executors.status_handling INFO: Allocated block ID 2799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2799 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2799 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2799.1709793937.6557972.submit parsl.executors.status_handling INFO: Allocated block ID 2800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2800 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2800 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2800.1709793937.6563716.submit parsl.executors.status_handling INFO: Allocated block ID 2801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2801 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2801 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2801.1709793937.6568174.submit parsl.executors.status_handling INFO: Allocated block ID 2802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2802 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2802 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2802.1709793937.6572542.submit parsl.executors.status_handling INFO: Allocated block ID 2803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2803 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2803 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2803.1709793937.6577008.submit parsl.executors.status_handling INFO: Allocated block ID 2804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2804 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2804 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2804.1709793937.6581461.submit parsl.executors.status_handling INFO: Allocated block ID 2805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2805 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2805 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2805.1709793937.6585815.submit parsl.executors.status_handling INFO: Allocated block ID 2806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2806 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2806 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2806.1709793937.659023.submit parsl.executors.status_handling INFO: Allocated block ID 2807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2807 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2807 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2807.1709793937.6594577.submit parsl.executors.status_handling INFO: Allocated block ID 2808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2808 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2808 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2808.1709793937.6598914.submit parsl.executors.status_handling INFO: Allocated block ID 2809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2809 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2809 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2809.1709793937.66032.submit parsl.executors.status_handling INFO: Allocated block ID 2810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2810 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2810 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2810.1709793937.6607497.submit parsl.executors.status_handling INFO: Allocated block ID 2811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2811 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2811 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2811.1709793937.6611845.submit parsl.executors.status_handling INFO: Allocated block ID 2812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2812 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2812 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2812.1709793937.661625.submit parsl.executors.status_handling INFO: Allocated block ID 2813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2813 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2813 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2813.1709793937.662066.submit parsl.executors.status_handling INFO: Allocated block ID 2814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2814 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2814 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2814.1709793937.6624947.submit parsl.executors.status_handling INFO: Allocated block ID 2815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2815 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2815 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2815.1709793937.6629417.submit parsl.executors.status_handling INFO: Allocated block ID 2816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2816 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2816 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2816.1709793937.6633763.submit parsl.executors.status_handling INFO: Allocated block ID 2817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2817 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2817 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2817.1709793937.6638167.submit parsl.executors.status_handling INFO: Allocated block ID 2818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2818 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2818 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2818.1709793937.664252.submit parsl.executors.status_handling INFO: Allocated block ID 2819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2819 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2819 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2819.1709793937.6646743.submit parsl.executors.status_handling INFO: Allocated block ID 2820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2820 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2820 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2820.1709793937.6650994.submit parsl.executors.status_handling INFO: Allocated block ID 2821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2821 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2821 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2821.1709793937.6655288.submit parsl.executors.status_handling INFO: Allocated block ID 2822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2822 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2822 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2822.1709793937.6660836.submit parsl.executors.status_handling INFO: Allocated block ID 2823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2823 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2823 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2823.1709793937.6665452.submit parsl.executors.status_handling INFO: Allocated block ID 2824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2824 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2824 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2824.1709793937.666989.submit parsl.executors.status_handling INFO: Allocated block ID 2825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2825 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2825 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2825.1709793937.6674216.submit parsl.executors.status_handling INFO: Allocated block ID 2826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2826 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2826 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2826.1709793937.6678603.submit parsl.executors.status_handling INFO: Allocated block ID 2827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2827 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2827 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2827.1709793937.6682973.submit parsl.executors.status_handling INFO: Allocated block ID 2828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2828 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2828 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2828.1709793937.668728.submit parsl.executors.status_handling INFO: Allocated block ID 2829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2829 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2829 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2829.1709793937.6691685.submit parsl.executors.status_handling INFO: Allocated block ID 2830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2830 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2830 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2830.1709793937.6696026.submit parsl.executors.status_handling INFO: Allocated block ID 2831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2831 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2831 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2831.1709793937.6700413.submit parsl.executors.status_handling INFO: Allocated block ID 2832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2832 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2832 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2832.1709793937.670477.submit parsl.executors.status_handling INFO: Allocated block ID 2833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2833 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2833 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2833.1709793937.6709266.submit parsl.executors.status_handling INFO: Allocated block ID 2834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2834 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2834 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2834.1709793937.6713583.submit parsl.executors.status_handling INFO: Allocated block ID 2835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2835 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2835 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2835.1709793937.67179.submit parsl.executors.status_handling INFO: Allocated block ID 2836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2836 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2836 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2836.1709793937.6722286.submit parsl.executors.status_handling INFO: Allocated block ID 2837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2837 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2837 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2837.1709793937.672659.submit parsl.executors.status_handling INFO: Allocated block ID 2838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2838 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2838 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2838.1709793937.6730971.submit parsl.executors.status_handling INFO: Allocated block ID 2839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2839 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2839 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2839.1709793937.6735358.submit parsl.executors.status_handling INFO: Allocated block ID 2840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2840 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2840 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2840.1709793937.6739802.submit parsl.executors.status_handling INFO: Allocated block ID 2841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2841 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2841 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2841.1709793937.6744127.submit parsl.executors.status_handling INFO: Allocated block ID 2842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2842 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2842 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2842.1709793937.67485.submit parsl.executors.status_handling INFO: Allocated block ID 2843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2843 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2843 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2843.1709793937.6752732.submit parsl.executors.status_handling INFO: Allocated block ID 2844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2844 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2844 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2844.1709793937.6756942.submit parsl.executors.status_handling INFO: Allocated block ID 2845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2845 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2845 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2845.1709793937.6851265.submit parsl.executors.status_handling INFO: Allocated block ID 2846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2846 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2846 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2846.1709793937.685595.submit parsl.executors.status_handling INFO: Allocated block ID 2847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2847 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2847 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2847.1709793937.6860962.submit parsl.executors.status_handling INFO: Allocated block ID 2848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2848 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2848 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2848.1709793937.6865444.submit parsl.executors.status_handling INFO: Allocated block ID 2849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2849 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2849 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2849.1709793937.6869853.submit parsl.executors.status_handling INFO: Allocated block ID 2850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2850 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2850 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2850.1709793937.6874185.submit parsl.executors.status_handling INFO: Allocated block ID 2851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2851 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2851 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2851.1709793937.6878638.submit parsl.executors.status_handling INFO: Allocated block ID 2852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2852 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2852 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2852.1709793937.6882951.submit parsl.executors.status_handling INFO: Allocated block ID 2853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2853 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2853 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2853.1709793937.6887171.submit parsl.executors.status_handling INFO: Allocated block ID 2854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2854 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2854 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2854.1709793937.6891723.submit parsl.executors.status_handling INFO: Allocated block ID 2855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2855 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2855 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2855.1709793937.6896114.submit parsl.executors.status_handling INFO: Allocated block ID 2856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2856 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2856 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2856.1709793937.6900635.submit parsl.executors.status_handling INFO: Allocated block ID 2857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2857 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2857 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2857.1709793937.6905131.submit parsl.executors.status_handling INFO: Allocated block ID 2858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2858 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2858 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2858.1709793937.6909466.submit parsl.executors.status_handling INFO: Allocated block ID 2859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2859 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2859 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2859.1709793937.6913748.submit parsl.executors.status_handling INFO: Allocated block ID 2860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2860 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2860 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2860.1709793937.6918054.submit parsl.executors.status_handling INFO: Allocated block ID 2861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2861 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2861 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2861.1709793937.692243.submit parsl.executors.status_handling INFO: Allocated block ID 2862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2862 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2862 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2862.1709793937.6926665.submit parsl.executors.status_handling INFO: Allocated block ID 2863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2863 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2863 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2863.1709793937.6931047.submit parsl.executors.status_handling INFO: Allocated block ID 2864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2864 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2864 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2864.1709793937.6935303.submit parsl.executors.status_handling INFO: Allocated block ID 2865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2865 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2865 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2865.1709793937.6939597.submit parsl.executors.status_handling INFO: Allocated block ID 2866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2866 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2866 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2866.1709793937.6943898.submit parsl.executors.status_handling INFO: Allocated block ID 2867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2867 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2867 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2867.1709793937.6948178.submit parsl.executors.status_handling INFO: Allocated block ID 2868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2868 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2868 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2868.1709793937.70424.submit parsl.executors.status_handling INFO: Allocated block ID 2869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2869 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2869 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2869.1709793937.7046916.submit parsl.executors.status_handling INFO: Allocated block ID 2870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2870 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2870 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2870.1709793937.7052338.submit parsl.executors.status_handling INFO: Allocated block ID 2871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2871 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2871 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2871.1709793937.705679.submit parsl.executors.status_handling INFO: Allocated block ID 2872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2872 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2872 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2872.1709793937.7061272.submit parsl.executors.status_handling INFO: Allocated block ID 2873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2873 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2873 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2873.1709793937.706561.submit parsl.executors.status_handling INFO: Allocated block ID 2874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2874 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2874 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2874.1709793937.7069955.submit parsl.executors.status_handling INFO: Allocated block ID 2875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2875 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2875 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2875.1709793937.707437.submit parsl.executors.status_handling INFO: Allocated block ID 2876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2876 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2876 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2876.1709793937.7078874.submit parsl.executors.status_handling INFO: Allocated block ID 2877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2877 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2877 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2877.1709793937.7083285.submit parsl.executors.status_handling INFO: Allocated block ID 2878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2878 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2878 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2878.1709793937.7087653.submit parsl.executors.status_handling INFO: Allocated block ID 2879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2879 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2879 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2879.1709793937.7092075.submit parsl.executors.status_handling INFO: Allocated block ID 2880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2880 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2880 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2880.1709793937.7096312.submit parsl.executors.status_handling INFO: Allocated block ID 2881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2881 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2881 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2881.1709793937.7100608.submit parsl.executors.status_handling INFO: Allocated block ID 2882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2882 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2882 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2882.1709793937.7104857.submit parsl.executors.status_handling INFO: Allocated block ID 2883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2883 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2883 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2883.1709793937.7109156.submit parsl.executors.status_handling INFO: Allocated block ID 2884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2884 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2884 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2884.1709793937.711346.submit parsl.executors.status_handling INFO: Allocated block ID 2885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2885 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2885 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2885.1709793937.7117708.submit parsl.executors.status_handling INFO: Allocated block ID 2886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2886 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2886 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2886.1709793937.7122066.submit parsl.executors.status_handling INFO: Allocated block ID 2887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2887 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2887 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2887.1709793937.7126448.submit parsl.executors.status_handling INFO: Allocated block ID 2888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2888 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2888 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2888.1709793937.713083.submit parsl.executors.status_handling INFO: Allocated block ID 2889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2889 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2889 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2889.1709793937.7135143.submit parsl.executors.status_handling INFO: Allocated block ID 2890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2890 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2890 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2890.1709793937.7229772.submit parsl.executors.status_handling INFO: Allocated block ID 2891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2891 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2891 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2891.1709793937.7234821.submit parsl.executors.status_handling INFO: Allocated block ID 2892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2892 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2892 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2892.1709793937.7239356.submit parsl.executors.status_handling INFO: Allocated block ID 2893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2893 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2893 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2893.1709793937.7243776.submit parsl.executors.status_handling INFO: Allocated block ID 2894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2894 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2894 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2894.1709793937.724819.submit parsl.executors.status_handling INFO: Allocated block ID 2895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2895 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2895 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2895.1709793937.7252533.submit parsl.executors.status_handling INFO: Allocated block ID 2896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2896 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2896 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2896.1709793937.7256892.submit parsl.executors.status_handling INFO: Allocated block ID 2897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2897 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2897 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2897.1709793937.726142.submit parsl.executors.status_handling INFO: Allocated block ID 2898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2898 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2898 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2898.1709793937.7265859.submit parsl.executors.status_handling INFO: Allocated block ID 2899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2899 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2899 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2899.1709793937.7270386.submit parsl.executors.status_handling INFO: Allocated block ID 2900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2900 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2900 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2900.1709793937.727475.submit parsl.executors.status_handling INFO: Allocated block ID 2901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2901 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2901 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2901.1709793937.727916.submit parsl.executors.status_handling INFO: Allocated block ID 2902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2902 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2902 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2902.1709793937.7283502.submit parsl.executors.status_handling INFO: Allocated block ID 2903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2903 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2903 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2903.1709793937.7287793.submit parsl.executors.status_handling INFO: Allocated block ID 2904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2904 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2904 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2904.1709793937.729221.submit parsl.executors.status_handling INFO: Allocated block ID 2905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2905 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2905 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2905.1709793937.7296534.submit parsl.executors.status_handling INFO: Allocated block ID 2906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2906 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2906 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2906.1709793937.7300854.submit parsl.executors.status_handling INFO: Allocated block ID 2907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2907 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2907 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2907.1709793937.7305152.submit parsl.executors.status_handling INFO: Allocated block ID 2908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2908 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2908 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2908.1709793937.7309492.submit parsl.executors.status_handling INFO: Allocated block ID 2909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2909 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2909 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2909.1709793937.7313755.submit parsl.executors.status_handling INFO: Allocated block ID 2910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2910 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2910 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2910.1709793937.731814.submit parsl.executors.status_handling INFO: Allocated block ID 2911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2911 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2911 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2911.1709793937.7322538.submit parsl.executors.status_handling INFO: Allocated block ID 2912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2912 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2912 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2912.1709793937.7326846.submit parsl.executors.status_handling INFO: Allocated block ID 2913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2913 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2913 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2913.1709793937.733119.submit parsl.executors.status_handling INFO: Allocated block ID 2914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2914 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2914 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2914.1709793937.7335455.submit parsl.executors.status_handling INFO: Allocated block ID 2915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2915 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2915 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2915.1709793937.733978.submit parsl.executors.status_handling INFO: Allocated block ID 2916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2916 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2916 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2916.1709793937.7344036.submit parsl.executors.status_handling INFO: Allocated block ID 2917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2917 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2917 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2917.1709793937.7348385.submit parsl.executors.status_handling INFO: Allocated block ID 2918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2918 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2918 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2918.1709793937.7352757.submit parsl.executors.status_handling INFO: Allocated block ID 2919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2919 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2919 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2919.1709793937.7357013.submit parsl.executors.status_handling INFO: Allocated block ID 2920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2920 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2920 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2920.1709793937.736134.submit parsl.executors.status_handling INFO: Allocated block ID 2921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2921 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2921 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2921.1709793937.736561.submit parsl.executors.status_handling INFO: Allocated block ID 2922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2922 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2922 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2922.1709793937.7369912.submit parsl.executors.status_handling INFO: Allocated block ID 2923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2923 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2923 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2923.1709793937.7374268.submit parsl.executors.status_handling INFO: Allocated block ID 2924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2924 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2924 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2924.1709793937.737864.submit parsl.executors.status_handling INFO: Allocated block ID 2925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2925 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2925 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2925.1709793937.7382967.submit parsl.executors.status_handling INFO: Allocated block ID 2926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2926 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2926 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2926.1709793937.7387333.submit parsl.executors.status_handling INFO: Allocated block ID 2927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2927 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2927 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2927.1709793937.7391708.submit parsl.executors.status_handling INFO: Allocated block ID 2928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2928 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2928 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2928.1709793937.739601.submit parsl.executors.status_handling INFO: Allocated block ID 2929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2929 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2929 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2929.1709793937.7400348.submit parsl.executors.status_handling INFO: Allocated block ID 2930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2930 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2930 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2930.1709793937.7404904.submit parsl.executors.status_handling INFO: Allocated block ID 2931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2931 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2931 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2931.1709793937.7540107.submit parsl.executors.status_handling INFO: Allocated block ID 2932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2932 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2932 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2932.1709793937.754485.submit parsl.executors.status_handling INFO: Allocated block ID 2933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2933 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2933 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2933.1709793937.7549305.submit parsl.executors.status_handling INFO: Allocated block ID 2934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2934 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2934 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2934.1709793937.7553675.submit parsl.executors.status_handling INFO: Allocated block ID 2935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2935 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2935 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2935.1709793937.755813.submit parsl.executors.status_handling INFO: Allocated block ID 2936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2936 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2936 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2936.1709793937.7562516.submit parsl.executors.status_handling INFO: Allocated block ID 2937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2937 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2937 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2937.1709793937.756691.submit parsl.executors.status_handling INFO: Allocated block ID 2938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2938 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2938 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2938.1709793937.7571406.submit parsl.executors.status_handling INFO: Allocated block ID 2939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2939 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2939 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2939.1709793937.7575815.submit parsl.executors.status_handling INFO: Allocated block ID 2940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2940 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2940 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2940.1709793937.7580266.submit parsl.executors.status_handling INFO: Allocated block ID 2941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2941 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2941 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2941.1709793937.758461.submit parsl.executors.status_handling INFO: Allocated block ID 2942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2942 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2942 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2942.1709793937.7589028.submit parsl.executors.status_handling INFO: Allocated block ID 2943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2943 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2943 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2943.1709793937.7593338.submit parsl.executors.status_handling INFO: Allocated block ID 2944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2944 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2944 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2944.1709793937.7597704.submit parsl.executors.status_handling INFO: Allocated block ID 2945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2945 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2945 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2945.1709793937.7602024.submit parsl.executors.status_handling INFO: Allocated block ID 2946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2946 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2946 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2946.1709793937.7606342.submit parsl.executors.status_handling INFO: Allocated block ID 2947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2947 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2947 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2947.1709793937.7610838.submit parsl.executors.status_handling INFO: Allocated block ID 2948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2948 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2948 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2948.1709793937.7615206.submit parsl.executors.status_handling INFO: Allocated block ID 2949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2949 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2949 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2949.1709793937.7619603.submit parsl.executors.status_handling INFO: Allocated block ID 2950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2950 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2950 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2950.1709793937.7623975.submit parsl.executors.status_handling INFO: Allocated block ID 2951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2951 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2951 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2951.1709793937.7628357.submit parsl.executors.status_handling INFO: Allocated block ID 2952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2952 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2952 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2952.1709793937.7632716.submit parsl.executors.status_handling INFO: Allocated block ID 2953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2953 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2953 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2953.1709793937.7637067.submit parsl.executors.status_handling INFO: Allocated block ID 2954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2954 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2954 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2954.1709793937.7641373.submit parsl.executors.status_handling INFO: Allocated block ID 2955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2955 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2955 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2955.1709793937.764563.submit parsl.executors.status_handling INFO: Allocated block ID 2956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2956 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2956 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2956.1709793937.765002.submit parsl.executors.status_handling INFO: Allocated block ID 2957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2957 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2957 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2957.1709793937.7654288.submit parsl.executors.status_handling INFO: Allocated block ID 2958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2958 2024-03-06 22:45:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2958 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2958.1709793937.765861.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1350 parsl.dataflow.strategy DEBUG: Executor multi has 1350 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 2959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2959 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2959 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2959.1709793942.6501503.submit parsl.executors.status_handling INFO: Allocated block ID 2960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2960 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2960 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2960.1709793942.6506453.submit parsl.executors.status_handling INFO: Allocated block ID 2961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2961 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2961 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2961.1709793942.6511004.submit parsl.executors.status_handling INFO: Allocated block ID 2962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2962 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2962 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2962.1709793942.651533.submit parsl.executors.status_handling INFO: Allocated block ID 2963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2963 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2963 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2963.1709793942.6519876.submit parsl.executors.status_handling INFO: Allocated block ID 2964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2964 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2964 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2964.1709793942.652429.submit parsl.executors.status_handling INFO: Allocated block ID 2965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2965 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2965 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2965.1709793942.6528816.submit parsl.executors.status_handling INFO: Allocated block ID 2966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2966 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2966 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2966.1709793942.6533248.submit parsl.executors.status_handling INFO: Allocated block ID 2967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2967 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2967 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2967.1709793942.6537685.submit parsl.executors.status_handling INFO: Allocated block ID 2968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2968 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2968 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2968.1709793942.654215.submit parsl.executors.status_handling INFO: Allocated block ID 2969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2969 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2969 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2969.1709793942.6546502.submit parsl.executors.status_handling INFO: Allocated block ID 2970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2970 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2970 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2970.1709793942.6551003.submit parsl.executors.status_handling INFO: Allocated block ID 2971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2971 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2971 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2971.1709793942.655542.submit parsl.executors.status_handling INFO: Allocated block ID 2972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2972 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2972 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2972.1709793942.656017.submit parsl.executors.status_handling INFO: Allocated block ID 2973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2973 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2973 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2973.1709793942.6564784.submit parsl.executors.status_handling INFO: Allocated block ID 2974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2974 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2974 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2974.1709793942.656953.submit parsl.executors.status_handling INFO: Allocated block ID 2975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2975 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2975 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2975.1709793942.6573899.submit parsl.executors.status_handling INFO: Allocated block ID 2976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2976 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2976 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2976.1709793942.657828.submit parsl.executors.status_handling INFO: Allocated block ID 2977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2977 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2977 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2977.1709793942.6582668.submit parsl.executors.status_handling INFO: Allocated block ID 2978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2978 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2978 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2978.1709793942.6586928.submit parsl.executors.status_handling INFO: Allocated block ID 2979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2979 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2979 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2979.1709793942.659145.submit parsl.executors.status_handling INFO: Allocated block ID 2980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2980 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2980 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2980.1709793942.6595788.submit parsl.executors.status_handling INFO: Allocated block ID 2981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2981 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2981 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2981.1709793942.6600173.submit parsl.executors.status_handling INFO: Allocated block ID 2982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2982 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2982 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2982.1709793942.6604698.submit parsl.executors.status_handling INFO: Allocated block ID 2983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2983 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2983 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2983.1709793942.6609228.submit parsl.executors.status_handling INFO: Allocated block ID 2984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2984 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2984 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2984.1709793942.6613653.submit parsl.executors.status_handling INFO: Allocated block ID 2985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2985 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2985 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2985.1709793942.6618018.submit parsl.executors.status_handling INFO: Allocated block ID 2986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2986 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2986 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2986.1709793942.6733632.submit parsl.executors.status_handling INFO: Allocated block ID 2987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2987 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2987 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2987.1709793942.673804.submit parsl.executors.status_handling INFO: Allocated block ID 2988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2988 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2988 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2988.1709793942.6742525.submit parsl.executors.status_handling INFO: Allocated block ID 2989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2989 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2989 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2989.1709793942.6746795.submit parsl.executors.status_handling INFO: Allocated block ID 2990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2990 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2990 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2990.1709793942.6751158.submit parsl.executors.status_handling INFO: Allocated block ID 2991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2991 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2991 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2991.1709793942.6755698.submit parsl.executors.status_handling INFO: Allocated block ID 2992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2992 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2992 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2992.1709793942.676015.submit parsl.executors.status_handling INFO: Allocated block ID 2993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2993 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2993 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2993.1709793942.6764586.submit parsl.executors.status_handling INFO: Allocated block ID 2994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2994 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2994 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2994.1709793942.6769173.submit parsl.executors.status_handling INFO: Allocated block ID 2995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2995 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2995 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2995.1709793942.6773598.submit parsl.executors.status_handling INFO: Allocated block ID 2996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2996 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2996 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2996.1709793942.6778004.submit parsl.executors.status_handling INFO: Allocated block ID 2997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2997 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2997 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2997.1709793942.6782596.submit parsl.executors.status_handling INFO: Allocated block ID 2998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2998 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2998 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2998.1709793942.6786857.submit parsl.executors.status_handling INFO: Allocated block ID 2999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2999 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2999 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=2999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-2999.1709793942.6791193.submit parsl.executors.status_handling INFO: Allocated block ID 3000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3000 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3000 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3000.1709793942.6795437.submit parsl.executors.status_handling INFO: Allocated block ID 3001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3001 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3001 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3001.1709793942.679997.submit parsl.executors.status_handling INFO: Allocated block ID 3002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3002 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3002 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3002.1709793942.6804364.submit parsl.executors.status_handling INFO: Allocated block ID 3003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3003 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3003 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3003.1709793942.6808872.submit parsl.executors.status_handling INFO: Allocated block ID 3004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3004 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3004 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3004.1709793942.6813247.submit parsl.executors.status_handling INFO: Allocated block ID 3005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3005 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3005 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3005.1709793942.6817505.submit parsl.executors.status_handling INFO: Allocated block ID 3006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3006 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3006 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3006.1709793942.6822143.submit parsl.executors.status_handling INFO: Allocated block ID 3007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3007 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3007 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3007.1709793942.682656.submit parsl.executors.status_handling INFO: Allocated block ID 3008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3008 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3008 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3008.1709793942.6831055.submit parsl.executors.status_handling INFO: Allocated block ID 3009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3009 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3009 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3009.1709793942.6835425.submit parsl.executors.status_handling INFO: Allocated block ID 3010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3010 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3010 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3010.1709793942.6839933.submit parsl.executors.status_handling INFO: Allocated block ID 3011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3011 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3011 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3011.1709793942.684423.submit parsl.executors.status_handling INFO: Allocated block ID 3012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3012 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3012 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3012.1709793942.6848626.submit parsl.executors.status_handling INFO: Allocated block ID 3013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3013 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3013 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3013.1709793942.6852875.submit parsl.executors.status_handling INFO: Allocated block ID 3014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3014 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3014 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3014.1709793942.6857138.submit parsl.executors.status_handling INFO: Allocated block ID 3015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3015 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3015 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3015.1709793942.6861413.submit parsl.executors.status_handling INFO: Allocated block ID 3016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3016 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3016 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3016.1709793942.686562.submit parsl.executors.status_handling INFO: Allocated block ID 3017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3017 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3017 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3017.1709793942.686989.submit parsl.executors.status_handling INFO: Allocated block ID 3018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3018 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3018 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3018.1709793942.6874163.submit parsl.executors.status_handling INFO: Allocated block ID 3019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3019 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3019 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3019.1709793942.6878514.submit parsl.executors.status_handling INFO: Allocated block ID 3020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3020 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3020 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3020.1709793942.6882784.submit parsl.executors.status_handling INFO: Allocated block ID 3021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3021 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3021 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3021.1709793942.6887014.submit parsl.executors.status_handling INFO: Allocated block ID 3022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3022 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3022 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3022.1709793942.6891322.submit parsl.executors.status_handling INFO: Allocated block ID 3023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3023 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3023 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3023.1709793942.6895578.submit parsl.executors.status_handling INFO: Allocated block ID 3024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3024 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3024 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3024.1709793942.689993.submit parsl.executors.status_handling INFO: Allocated block ID 3025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3025 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3025 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3025.1709793942.6904185.submit parsl.executors.status_handling INFO: Allocated block ID 3026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3026 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3026 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3026.1709793942.6909878.submit parsl.executors.status_handling INFO: Allocated block ID 3027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3027 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3027 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3027.1709793942.6914475.submit parsl.executors.status_handling INFO: Allocated block ID 3028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3028 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3028 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3028.1709793942.6918862.submit parsl.executors.status_handling INFO: Allocated block ID 3029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3029 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3029 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3029.1709793942.6923158.submit parsl.executors.status_handling INFO: Allocated block ID 3030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3030 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3030 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3030.1709793942.6927462.submit parsl.executors.status_handling INFO: Allocated block ID 3031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3031 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3031 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3031.1709793942.6932127.submit parsl.executors.status_handling INFO: Allocated block ID 3032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3032 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3032 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3032.1709793942.6936572.submit parsl.executors.status_handling INFO: Allocated block ID 3033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3033 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3033 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3033.1709793942.6946027.submit parsl.executors.status_handling INFO: Allocated block ID 3034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3034 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3034 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3034.1709793942.6981344.submit parsl.executors.status_handling INFO: Allocated block ID 3035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3035 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3035 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3035.1709793942.6987004.submit parsl.executors.status_handling INFO: Allocated block ID 3036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3036 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3036 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3036.1709793942.6992402.submit parsl.executors.status_handling INFO: Allocated block ID 3037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3037 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3037 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3037.1709793942.6997604.submit parsl.executors.status_handling INFO: Allocated block ID 3038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3038 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3038 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3038.1709793942.7002892.submit parsl.executors.status_handling INFO: Allocated block ID 3039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3039 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3039 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3039.1709793942.7008142.submit parsl.executors.status_handling INFO: Allocated block ID 3040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3040 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3040 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3040.1709793942.701333.submit parsl.executors.status_handling INFO: Allocated block ID 3041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3041 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3041 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3041.1709793942.7018638.submit parsl.executors.status_handling INFO: Allocated block ID 3042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3042 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3042 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3042.1709793942.7023895.submit parsl.executors.status_handling INFO: Allocated block ID 3043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3043 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3043 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3043.1709793942.7029188.submit parsl.executors.status_handling INFO: Allocated block ID 3044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3044 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3044 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3044.1709793942.7034345.submit parsl.executors.status_handling INFO: Allocated block ID 3045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3045 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3045 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3045.1709793942.7039533.submit parsl.executors.status_handling INFO: Allocated block ID 3046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3046 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3046 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3046.1709793942.7044256.submit parsl.executors.status_handling INFO: Allocated block ID 3047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3047 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3047 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3047.1709793942.7048924.submit parsl.executors.status_handling INFO: Allocated block ID 3048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3048 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3048 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3048.1709793942.70535.submit parsl.executors.status_handling INFO: Allocated block ID 3049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3049 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3049 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3049.1709793942.7058039.submit parsl.executors.status_handling INFO: Allocated block ID 3050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3050 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3050 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3050.1709793942.7062693.submit parsl.executors.status_handling INFO: Allocated block ID 3051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3051 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3051 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3051.1709793942.7067254.submit parsl.executors.status_handling INFO: Allocated block ID 3052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3052 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3052 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3052.1709793942.7072105.submit parsl.executors.status_handling INFO: Allocated block ID 3053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3053 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3053 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3053.1709793942.7076657.submit parsl.executors.status_handling INFO: Allocated block ID 3054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3054 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3054 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3054.1709793942.7081354.submit parsl.executors.status_handling INFO: Allocated block ID 3055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3055 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3055 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3055.1709793942.7085927.submit parsl.executors.status_handling INFO: Allocated block ID 3056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3056 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3056 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3056.1709793942.7152412.submit parsl.executors.status_handling INFO: Allocated block ID 3057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3057 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3057 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3057.1709793942.7158005.submit parsl.executors.status_handling INFO: Allocated block ID 3058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3058 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3058 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3058.1709793942.716334.submit parsl.executors.status_handling INFO: Allocated block ID 3059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3059 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3059 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3059.1709793942.7168605.submit parsl.executors.status_handling INFO: Allocated block ID 3060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3060 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3060 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3060.1709793942.7173734.submit parsl.executors.status_handling INFO: Allocated block ID 3061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3061 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3061 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3061.1709793942.7178957.submit parsl.executors.status_handling INFO: Allocated block ID 3062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3062 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3062 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3062.1709793942.7184174.submit parsl.executors.status_handling INFO: Allocated block ID 3063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3063 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3063 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3063.1709793942.7189455.submit parsl.executors.status_handling INFO: Allocated block ID 3064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3064 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3064 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3064.1709793942.7194622.submit parsl.executors.status_handling INFO: Allocated block ID 3065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3065 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3065 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3065.1709793942.7199872.submit parsl.executors.status_handling INFO: Allocated block ID 3066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3066 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3066 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3066.1709793942.720512.submit parsl.executors.status_handling INFO: Allocated block ID 3067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3067 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3067 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3067.1709793942.721047.submit parsl.executors.status_handling INFO: Allocated block ID 3068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3068 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3068 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3068.1709793942.7215707.submit parsl.executors.status_handling INFO: Allocated block ID 3069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3069 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3069 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3069.1709793942.7220972.submit parsl.executors.status_handling INFO: Allocated block ID 3070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3070 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3070 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3070.1709793942.7226083.submit parsl.executors.status_handling INFO: Allocated block ID 3071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3071 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3071 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3071.1709793942.723133.submit parsl.executors.status_handling INFO: Allocated block ID 3072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3072 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3072 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3072.1709793942.7236977.submit parsl.executors.status_handling INFO: Allocated block ID 3073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3073 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3073 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3073.1709793942.724247.submit parsl.executors.status_handling INFO: Allocated block ID 3074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3074 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3074 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3074.1709793942.7247643.submit parsl.executors.status_handling INFO: Allocated block ID 3075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3075 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3075 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3075.1709793942.7252915.submit parsl.executors.status_handling INFO: Allocated block ID 3076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3076 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3076 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3076.1709793942.739908.submit parsl.executors.status_handling INFO: Allocated block ID 3077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3077 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3077 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3077.1709793942.740544.submit parsl.executors.status_handling INFO: Allocated block ID 3078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3078 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3078 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3078.1709793942.74109.submit parsl.executors.status_handling INFO: Allocated block ID 3079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3079 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3079 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3079.1709793942.7416081.submit parsl.executors.status_handling INFO: Allocated block ID 3080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3080 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3080 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3080.1709793942.7421355.submit parsl.executors.status_handling INFO: Allocated block ID 3081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3081 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3081 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3081.1709793942.7426488.submit parsl.executors.status_handling INFO: Allocated block ID 3082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3082 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3082 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3082.1709793942.7431667.submit parsl.executors.status_handling INFO: Allocated block ID 3083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3083 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3083 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3083.1709793942.7436216.submit parsl.executors.status_handling INFO: Allocated block ID 3084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3084 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3084 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3084.1709793942.744119.submit parsl.executors.status_handling INFO: Allocated block ID 3085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3085 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3085 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3085.1709793942.7445753.submit parsl.executors.status_handling INFO: Allocated block ID 3086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3086 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3086 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3086.1709793942.7450402.submit parsl.executors.status_handling INFO: Allocated block ID 3087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3087 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3087 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3087.1709793942.745503.submit parsl.executors.status_handling INFO: Allocated block ID 3088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3088 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3088 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3088.1709793942.74597.submit parsl.executors.status_handling INFO: Allocated block ID 3089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3089 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3089 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3089.1709793942.7464259.submit parsl.executors.status_handling INFO: Allocated block ID 3090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3090 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3090 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3090.1709793942.7468956.submit parsl.executors.status_handling INFO: Allocated block ID 3091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3091 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3091 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3091.1709793942.747352.submit parsl.executors.status_handling INFO: Allocated block ID 3092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3092 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3092 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3092.1709793942.7478185.submit parsl.executors.status_handling INFO: Allocated block ID 3093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3093 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3093 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3093.1709793942.748327.submit parsl.executors.status_handling INFO: Allocated block ID 3094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3094 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3094 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3094.1709793942.7580588.submit parsl.executors.status_handling INFO: Allocated block ID 3095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3095 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3095 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3095.1709793942.7586467.submit parsl.executors.status_handling INFO: Allocated block ID 3096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3096 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3096 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3096.1709793942.7591798.submit parsl.executors.status_handling INFO: Allocated block ID 3097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3097 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3097 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3097.1709793942.7597008.submit parsl.executors.status_handling INFO: Allocated block ID 3098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3098 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3098 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3098.1709793942.7602317.submit parsl.executors.status_handling INFO: Allocated block ID 3099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3099 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3099 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3099.1709793942.7607453.submit parsl.executors.status_handling INFO: Allocated block ID 3100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3100 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3100 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3100.1709793942.761264.submit parsl.executors.status_handling INFO: Allocated block ID 3101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3101 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3101 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3101.1709793942.7617779.submit parsl.executors.status_handling INFO: Allocated block ID 3102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3102 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3102 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3102.1709793942.7623105.submit parsl.executors.status_handling INFO: Allocated block ID 3103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3103 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3103 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3103.1709793942.7628403.submit parsl.executors.status_handling INFO: Allocated block ID 3104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3104 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3104 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3104.1709793942.7633665.submit parsl.executors.status_handling INFO: Allocated block ID 3105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3105 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3105 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3105.1709793942.7638965.submit parsl.executors.status_handling INFO: Allocated block ID 3106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3106 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3106 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3106.1709793942.7644205.submit parsl.executors.status_handling INFO: Allocated block ID 3107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3107 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3107 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3107.1709793942.7649584.submit parsl.executors.status_handling INFO: Allocated block ID 3108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3108 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3108 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3108.1709793942.7654848.submit parsl.executors.status_handling INFO: Allocated block ID 3109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3109 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3109 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3109.1709793942.7721515.submit parsl.executors.status_handling INFO: Allocated block ID 3110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3110 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3110 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3110.1709793942.7727134.submit parsl.executors.status_handling INFO: Allocated block ID 3111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3111 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3111 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3111.1709793942.77325.submit parsl.executors.status_handling INFO: Allocated block ID 3112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3112 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3112 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3112.1709793942.7737648.submit parsl.executors.status_handling INFO: Allocated block ID 3113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3113 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3113 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3113.1709793942.7742963.submit parsl.executors.status_handling INFO: Allocated block ID 3114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3114 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3114 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3114.1709793942.7748299.submit parsl.executors.status_handling INFO: Allocated block ID 3115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3115 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3115 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3115.1709793942.775351.submit parsl.executors.status_handling INFO: Allocated block ID 3116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3116 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3116 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3116.1709793942.7758808.submit parsl.executors.status_handling INFO: Allocated block ID 3117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3117 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3117 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3117.1709793942.7764053.submit parsl.executors.status_handling INFO: Allocated block ID 3118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3118 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3118 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3118.1709793942.7769287.submit parsl.executors.status_handling INFO: Allocated block ID 3119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3119 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3119 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3119.1709793942.7774458.submit parsl.executors.status_handling INFO: Allocated block ID 3120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3120 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3120 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3120.1709793942.7891967.submit parsl.executors.status_handling INFO: Allocated block ID 3121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3121 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3121 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3121.1709793942.7897687.submit parsl.executors.status_handling INFO: Allocated block ID 3122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3122 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3122 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3122.1709793942.7903092.submit parsl.executors.status_handling INFO: Allocated block ID 3123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3123 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3123 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3123.1709793942.7908425.submit parsl.executors.status_handling INFO: Allocated block ID 3124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3124 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3124 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3124.1709793942.7913659.submit parsl.executors.status_handling INFO: Allocated block ID 3125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3125 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3125 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3125.1709793942.7919064.submit parsl.executors.status_handling INFO: Allocated block ID 3126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3126 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3126 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3126.1709793942.7924407.submit parsl.executors.status_handling INFO: Allocated block ID 3127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3127 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3127 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3127.1709793942.7929766.submit parsl.executors.status_handling INFO: Allocated block ID 3128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3128 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3128 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3128.1709793942.7935061.submit parsl.executors.status_handling INFO: Allocated block ID 3129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3129 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3129 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3129.1709793942.794038.submit parsl.executors.status_handling INFO: Allocated block ID 3130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3130 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3130 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3130.1709793942.7945635.submit parsl.executors.status_handling INFO: Allocated block ID 3131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3131 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3131 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3131.1709793942.7951038.submit parsl.executors.status_handling INFO: Allocated block ID 3132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3132 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3132 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3132.1709793942.795633.submit parsl.executors.status_handling INFO: Allocated block ID 3133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3133 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3133 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3133.1709793942.7962866.submit parsl.executors.status_handling INFO: Allocated block ID 3134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3134 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3134 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3134.1709793942.796833.submit parsl.executors.status_handling INFO: Allocated block ID 3135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3135 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3135 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3135.1709793942.7973578.submit parsl.executors.status_handling INFO: Allocated block ID 3136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3136 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3136 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3136.1709793942.79789.submit parsl.executors.status_handling INFO: Allocated block ID 3137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3137 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3137 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3137.1709793942.7984128.submit parsl.executors.status_handling INFO: Allocated block ID 3138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3138 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3138 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3138.1709793942.7989476.submit parsl.executors.status_handling INFO: Allocated block ID 3139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3139 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3139 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3139.1709793942.799473.submit parsl.executors.status_handling INFO: Allocated block ID 3140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3140 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3140 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3140.1709793942.800009.submit parsl.executors.status_handling INFO: Allocated block ID 3141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3141 2024-03-06 22:45:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3141 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3141.1709793942.8005311.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 2024-03-06 22:45:47 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1350 parsl.dataflow.strategy DEBUG: Executor multi has 1350 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 3142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3142 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3142 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3142.1709793947.7641737.submit parsl.executors.status_handling INFO: Allocated block ID 3143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3143 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3143 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3143.1709793947.7646868.submit parsl.executors.status_handling INFO: Allocated block ID 3144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3144 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3144 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3144.1709793947.7651272.submit parsl.executors.status_handling INFO: Allocated block ID 3145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3145 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3145 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3145.1709793947.7655528.submit parsl.executors.status_handling INFO: Allocated block ID 3146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3146 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3146 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3146.1709793947.7659822.submit parsl.executors.status_handling INFO: Allocated block ID 3147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3147 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3147 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3147.1709793947.766411.submit parsl.executors.status_handling INFO: Allocated block ID 3148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3148 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3148 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3148.1709793947.7668505.submit parsl.executors.status_handling INFO: Allocated block ID 3149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3149 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3149 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3149.1709793947.767279.submit parsl.executors.status_handling INFO: Allocated block ID 3150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3150 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3150 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3150.1709793947.767702.submit parsl.executors.status_handling INFO: Allocated block ID 3151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3151 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3151 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3151.1709793947.7681313.submit parsl.executors.status_handling INFO: Allocated block ID 3152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3152 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3152 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3152.1709793947.7685528.submit parsl.executors.status_handling INFO: Allocated block ID 3153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3153 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3153 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3153.1709793947.7689836.submit parsl.executors.status_handling INFO: Allocated block ID 3154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3154 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3154 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3154.1709793947.7694082.submit parsl.executors.status_handling INFO: Allocated block ID 3155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3155 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3155 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3155.1709793947.7698705.submit parsl.executors.status_handling INFO: Allocated block ID 3156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3156 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3156 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3156.1709793947.7702994.submit parsl.executors.status_handling INFO: Allocated block ID 3157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3157 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3157 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3157.1709793947.7707207.submit parsl.executors.status_handling INFO: Allocated block ID 3158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3158 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3158 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3158.1709793947.7815073.submit parsl.executors.status_handling INFO: Allocated block ID 3159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3159 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3159 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3159.1709793947.7819595.submit parsl.executors.status_handling INFO: Allocated block ID 3160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3160 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3160 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3160.1709793947.7823918.submit parsl.executors.status_handling INFO: Allocated block ID 3161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3161 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3161 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3161.1709793947.7828188.submit parsl.executors.status_handling INFO: Allocated block ID 3162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3162 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3162 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3162.1709793947.7832458.submit parsl.executors.status_handling INFO: Allocated block ID 3163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3163 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3163 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3163.1709793947.7836668.submit parsl.executors.status_handling INFO: Allocated block ID 3164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3164 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3164 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3164.1709793947.7840948.submit parsl.executors.status_handling INFO: Allocated block ID 3165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3165 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3165 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3165.1709793947.7845132.submit parsl.executors.status_handling INFO: Allocated block ID 3166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3166 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3166 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3166.1709793947.784938.submit parsl.executors.status_handling INFO: Allocated block ID 3167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3167 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3167 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3167.1709793947.7853599.submit parsl.executors.status_handling INFO: Allocated block ID 3168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3168 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3168 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3168.1709793947.7857864.submit parsl.executors.status_handling INFO: Allocated block ID 3169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3169 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3169 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3169.1709793947.786216.submit parsl.executors.status_handling INFO: Allocated block ID 3170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3170 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3170 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3170.1709793947.786637.submit parsl.executors.status_handling INFO: Allocated block ID 3171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3171 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3171 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3171.1709793947.7870617.submit parsl.executors.status_handling INFO: Allocated block ID 3172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3172 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3172 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3172.1709793947.7874901.submit parsl.executors.status_handling INFO: Allocated block ID 3173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3173 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3173 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3173.1709793947.7879264.submit parsl.executors.status_handling INFO: Allocated block ID 3174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3174 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3174 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3174.1709793947.7883523.submit parsl.executors.status_handling INFO: Allocated block ID 3175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3175 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3175 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3175.1709793947.788772.submit parsl.executors.status_handling INFO: Allocated block ID 3176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3176 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3176 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3176.1709793947.7891977.submit parsl.executors.status_handling INFO: Allocated block ID 3177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3177 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3177 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3177.1709793947.7896233.submit parsl.executors.status_handling INFO: Allocated block ID 3178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3178 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3178 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3178.1709793947.7900534.submit parsl.executors.status_handling INFO: Allocated block ID 3179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3179 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3179 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3179.1709793947.790476.submit parsl.executors.status_handling INFO: Allocated block ID 3180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3180 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3180 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3180.1709793947.7909048.submit parsl.executors.status_handling INFO: Allocated block ID 3181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3181 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3181 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3181.1709793947.79133.submit parsl.executors.status_handling INFO: Allocated block ID 3182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3182 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3182 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3182.1709793947.79175.submit parsl.executors.status_handling INFO: Allocated block ID 3183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3183 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3183 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3183.1709793947.8023787.submit parsl.executors.status_handling INFO: Allocated block ID 3184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3184 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3184 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3184.1709793947.8028257.submit parsl.executors.status_handling INFO: Allocated block ID 3185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3185 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3185 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3185.1709793947.8032806.submit parsl.executors.status_handling INFO: Allocated block ID 3186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3186 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3186 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3186.1709793947.8037174.submit parsl.executors.status_handling INFO: Allocated block ID 3187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3187 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3187 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3187.1709793947.804181.submit parsl.executors.status_handling INFO: Allocated block ID 3188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3188 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3188 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3188.1709793947.804613.submit parsl.executors.status_handling INFO: Allocated block ID 3189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3189 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3189 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3189.1709793947.805063.submit parsl.executors.status_handling INFO: Allocated block ID 3190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3190 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3190 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3190.1709793947.8054833.submit parsl.executors.status_handling INFO: Allocated block ID 3191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3191 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3191 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3191.1709793947.8059213.submit parsl.executors.status_handling INFO: Allocated block ID 3192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3192 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3192 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3192.1709793947.8063633.submit parsl.executors.status_handling INFO: Allocated block ID 3193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3193 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3193 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3193.1709793947.8067954.submit parsl.executors.status_handling INFO: Allocated block ID 3194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3194 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3194 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3194.1709793947.80724.submit parsl.executors.status_handling INFO: Allocated block ID 3195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3195 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3195 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3195.1709793947.80767.submit parsl.executors.status_handling INFO: Allocated block ID 3196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3196 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3196 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3196.1709793947.80812.submit parsl.executors.status_handling INFO: Allocated block ID 3197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3197 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3197 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3197.1709793947.8085418.submit parsl.executors.status_handling INFO: Allocated block ID 3198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3198 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3198 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3198.1709793947.808972.submit parsl.executors.status_handling INFO: Allocated block ID 3199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3199 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3199 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3199.1709793947.80939.submit parsl.executors.status_handling INFO: Allocated block ID 3200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3200 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3200 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3200.1709793947.809813.submit parsl.executors.status_handling INFO: Allocated block ID 3201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3201 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3201 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3201.1709793947.8102393.submit parsl.executors.status_handling INFO: Allocated block ID 3202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3202 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3202 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3202.1709793947.8106546.submit parsl.executors.status_handling INFO: Allocated block ID 3203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3203 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3203 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3203.1709793947.8191907.submit parsl.executors.status_handling INFO: Allocated block ID 3204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3204 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3204 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3204.1709793947.8196483.submit parsl.executors.status_handling INFO: Allocated block ID 3205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3205 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3205 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3205.1709793947.8200848.submit parsl.executors.status_handling INFO: Allocated block ID 3206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3206 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3206 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3206.1709793947.8205097.submit parsl.executors.status_handling INFO: Allocated block ID 3207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3207 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3207 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3207.1709793947.8209426.submit parsl.executors.status_handling INFO: Allocated block ID 3208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3208 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3208 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3208.1709793947.8213718.submit parsl.executors.status_handling INFO: Allocated block ID 3209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3209 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3209 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3209.1709793947.821794.submit parsl.executors.status_handling INFO: Allocated block ID 3210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3210 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3210 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3210.1709793947.822231.submit parsl.executors.status_handling INFO: Allocated block ID 3211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3211 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3211 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3211.1709793947.8226526.submit parsl.executors.status_handling INFO: Allocated block ID 3212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3212 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3212 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3212.1709793947.8230762.submit parsl.executors.status_handling INFO: Allocated block ID 3213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3213 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3213 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3213.1709793947.8234954.submit parsl.executors.status_handling INFO: Allocated block ID 3214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3214 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3214 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3214.1709793947.8239179.submit parsl.executors.status_handling INFO: Allocated block ID 3215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3215 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3215 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3215.1709793947.824338.submit parsl.executors.status_handling INFO: Allocated block ID 3216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3216 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3216 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3216.1709793947.8247657.submit parsl.executors.status_handling INFO: Allocated block ID 3217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3217 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3217 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3217.1709793947.8251932.submit parsl.executors.status_handling INFO: Allocated block ID 3218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3218 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3218 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3218.1709793947.8256137.submit parsl.executors.status_handling INFO: Allocated block ID 3219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3219 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3219 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3219.1709793947.8260405.submit parsl.executors.status_handling INFO: Allocated block ID 3220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3220 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3220 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3220.1709793947.8264668.submit parsl.executors.status_handling INFO: Allocated block ID 3221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3221 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3221 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3221.1709793947.8268955.submit parsl.executors.status_handling INFO: Allocated block ID 3222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3222 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3222 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3222.1709793947.8273194.submit parsl.executors.status_handling INFO: Allocated block ID 3223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3223 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3223 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3223.1709793947.8277392.submit parsl.executors.status_handling INFO: Allocated block ID 3224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3224 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3224 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3224.1709793947.8282092.submit parsl.executors.status_handling INFO: Allocated block ID 3225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3225 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3225 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3225.1709793947.8286304.submit parsl.executors.status_handling INFO: Allocated block ID 3226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3226 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3226 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3226.1709793947.8290536.submit parsl.executors.status_handling INFO: Allocated block ID 3227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3227 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3227 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3227.1709793947.8294735.submit parsl.executors.status_handling INFO: Allocated block ID 3228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3228 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3228 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3228.1709793947.829905.submit parsl.executors.status_handling INFO: Allocated block ID 3229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3229 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3229 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3229.1709793947.8303213.submit parsl.executors.status_handling INFO: Allocated block ID 3230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3230 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3230 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3230.1709793947.8307395.submit parsl.executors.status_handling INFO: Allocated block ID 3231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3231 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3231 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3231.1709793947.8311605.submit parsl.executors.status_handling INFO: Allocated block ID 3232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3232 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3232 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3232.1709793947.8315856.submit parsl.executors.status_handling INFO: Allocated block ID 3233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3233 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3233 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3233.1709793947.83201.submit parsl.executors.status_handling INFO: Allocated block ID 3234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3234 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3234 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3234.1709793947.8324323.submit parsl.executors.status_handling INFO: Allocated block ID 3235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3235 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3235 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3235.1709793947.832855.submit parsl.executors.status_handling INFO: Allocated block ID 3236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3236 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3236 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3236.1709793947.8332756.submit parsl.executors.status_handling INFO: Allocated block ID 3237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3237 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3237 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3237.1709793947.8336964.submit parsl.executors.status_handling INFO: Allocated block ID 3238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3238 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3238 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3238.1709793947.8341267.submit parsl.executors.status_handling INFO: Allocated block ID 3239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3239 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3239 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3239.1709793947.8345478.submit parsl.executors.status_handling INFO: Allocated block ID 3240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3240 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3240 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3240.1709793947.8349788.submit parsl.executors.status_handling INFO: Allocated block ID 3241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3241 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3241 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3241.1709793947.8353992.submit parsl.executors.status_handling INFO: Allocated block ID 3242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3242 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3242 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3242.1709793947.8358238.submit parsl.executors.status_handling INFO: Allocated block ID 3243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3243 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3243 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3243.1709793947.8362713.submit parsl.executors.status_handling INFO: Allocated block ID 3244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3244 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3244 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3244.1709793947.836707.submit parsl.executors.status_handling INFO: Allocated block ID 3245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3245 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3245 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3245.1709793947.83715.submit parsl.executors.status_handling INFO: Allocated block ID 3246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3246 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3246 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3246.1709793947.8375723.submit parsl.executors.status_handling INFO: Allocated block ID 3247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3247 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3247 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3247.1709793947.8380034.submit parsl.executors.status_handling INFO: Allocated block ID 3248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3248 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3248 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3248.1709793947.8384254.submit parsl.executors.status_handling INFO: Allocated block ID 3249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3249 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3249 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3249.1709793947.8394601.submit parsl.executors.status_handling INFO: Allocated block ID 3250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3250 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3250 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3250.1709793947.8399718.submit parsl.executors.status_handling INFO: Allocated block ID 3251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3251 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3251 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3251.1709793947.8404133.submit parsl.executors.status_handling INFO: Allocated block ID 3252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3252 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3252 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3252.1709793947.8408535.submit parsl.executors.status_handling INFO: Allocated block ID 3253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3253 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3253 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3253.1709793947.8412895.submit parsl.executors.status_handling INFO: Allocated block ID 3254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3254 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3254 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3254.1709793947.841709.submit parsl.executors.status_handling INFO: Allocated block ID 3255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3255 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3255 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3255.1709793947.842136.submit parsl.executors.status_handling INFO: Allocated block ID 3256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3256 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3256 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3256.1709793947.84256.submit parsl.executors.status_handling INFO: Allocated block ID 3257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3257 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3257 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3257.1709793947.843016.submit parsl.executors.status_handling INFO: Allocated block ID 3258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3258 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3258 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3258.1709793947.8434474.submit parsl.executors.status_handling INFO: Allocated block ID 3259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3259 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3259 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3259.1709793947.843873.submit parsl.executors.status_handling INFO: Allocated block ID 3260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3260 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3260 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3260.1709793947.8442962.submit parsl.executors.status_handling INFO: Allocated block ID 3261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3261 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3261 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3261.1709793947.844721.submit parsl.executors.status_handling INFO: Allocated block ID 3262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3262 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3262 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3262.1709793947.84515.submit parsl.executors.status_handling INFO: Allocated block ID 3263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3263 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3263 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3263.1709793947.8455706.submit parsl.executors.status_handling INFO: Allocated block ID 3264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3264 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3264 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3264.1709793947.846003.submit parsl.executors.status_handling INFO: Allocated block ID 3265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3265 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3265 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3265.1709793947.8464236.submit parsl.executors.status_handling INFO: Allocated block ID 3266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3266 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3266 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3266.1709793947.8468525.submit parsl.executors.status_handling INFO: Allocated block ID 3267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3267 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3267 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3267.1709793947.8472767.submit parsl.executors.status_handling INFO: Allocated block ID 3268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3268 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3268 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3268.1709793947.8477023.submit parsl.executors.status_handling INFO: Allocated block ID 3269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3269 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3269 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3269.1709793947.848132.submit parsl.executors.status_handling INFO: Allocated block ID 3270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3270 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3270 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3270.1709793947.848554.submit parsl.executors.status_handling INFO: Allocated block ID 3271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3271 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3271 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3271.1709793947.8490255.submit parsl.executors.status_handling INFO: Allocated block ID 3272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3272 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3272 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3272.1709793947.8494563.submit parsl.executors.status_handling INFO: Allocated block ID 3273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3273 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3273 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3273.1709793947.8498864.submit parsl.executors.status_handling INFO: Allocated block ID 3274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3274 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3274 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3274.1709793947.8503098.submit parsl.executors.status_handling INFO: Allocated block ID 3275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3275 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3275 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3275.1709793947.8507328.submit parsl.executors.status_handling INFO: Allocated block ID 3276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3276 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3276 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3276.1709793947.8511622.submit parsl.executors.status_handling INFO: Allocated block ID 3277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3277 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3277 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3277.1709793947.851581.submit parsl.executors.status_handling INFO: Allocated block ID 3278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3278 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3278 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3278.1709793947.8520057.submit parsl.executors.status_handling INFO: Allocated block ID 3279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3279 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3279 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3279.1709793947.8524318.submit parsl.executors.status_handling INFO: Allocated block ID 3280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3280 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3280 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3280.1709793947.8528624.submit parsl.executors.status_handling INFO: Allocated block ID 3281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3281 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3281 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3281.1709793947.8532853.submit parsl.executors.status_handling INFO: Allocated block ID 3282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3282 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3282 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3282.1709793947.8537123.submit parsl.executors.status_handling INFO: Allocated block ID 3283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3283 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3283 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3283.1709793947.854146.submit parsl.executors.status_handling INFO: Allocated block ID 3284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3284 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3284 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3284.1709793947.8545668.submit parsl.executors.status_handling INFO: Allocated block ID 3285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3285 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3285 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3285.1709793947.8549922.submit parsl.executors.status_handling INFO: Allocated block ID 3286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3286 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3286 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3286.1709793947.8554103.submit parsl.executors.status_handling INFO: Allocated block ID 3287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3287 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3287 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3287.1709793947.8558352.submit parsl.executors.status_handling INFO: Allocated block ID 3288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3288 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3288 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3288.1709793947.8562605.submit parsl.executors.status_handling INFO: Allocated block ID 3289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3289 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3289 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3289.1709793947.85668.submit parsl.executors.status_handling INFO: Allocated block ID 3290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3290 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3290 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3290.1709793947.8571036.submit parsl.executors.status_handling INFO: Allocated block ID 3291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3291 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3291 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3291.1709793947.8575191.submit parsl.executors.status_handling INFO: Allocated block ID 3292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3292 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3292 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3292.1709793947.8579502.submit parsl.executors.status_handling INFO: Allocated block ID 3293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3293 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3293 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3293.1709793947.8583744.submit parsl.executors.status_handling INFO: Allocated block ID 3294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3294 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3294 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3294.1709793947.8587976.submit parsl.executors.status_handling INFO: Allocated block ID 3295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3295 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3295 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3295.1709793947.8646562.submit parsl.executors.status_handling INFO: Allocated block ID 3296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3296 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3296 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3296.1709793947.8651357.submit parsl.executors.status_handling INFO: Allocated block ID 3297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3297 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3297 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3297.1709793947.8655822.submit parsl.executors.status_handling INFO: Allocated block ID 3298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3298 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3298 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3298.1709793947.8660293.submit parsl.executors.status_handling INFO: Allocated block ID 3299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3299 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3299 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3299.1709793947.8664515.submit parsl.executors.status_handling INFO: Allocated block ID 3300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3300 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3300 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3300.1709793947.8668897.submit parsl.executors.status_handling INFO: Allocated block ID 3301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3301 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3301 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3301.1709793947.867316.submit parsl.executors.status_handling INFO: Allocated block ID 3302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3302 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3302 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3302.1709793947.867734.submit parsl.executors.status_handling INFO: Allocated block ID 3303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3303 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3303 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3303.1709793947.8681607.submit parsl.executors.status_handling INFO: Allocated block ID 3304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3304 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3304 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3304.1709793947.868588.submit parsl.executors.status_handling INFO: Allocated block ID 3305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3305 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3305 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3305.1709793947.869018.submit parsl.executors.status_handling INFO: Allocated block ID 3306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3306 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3306 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3306.1709793947.870122.submit parsl.executors.status_handling INFO: Allocated block ID 3307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3307 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3307 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3307.1709793947.8705828.submit parsl.executors.status_handling INFO: Allocated block ID 3308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3308 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3308 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3308.1709793947.8710382.submit parsl.executors.status_handling INFO: Allocated block ID 3309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3309 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3309 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3309.1709793947.871461.submit parsl.executors.status_handling INFO: Allocated block ID 3310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3310 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3310 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3310.1709793947.8718882.submit parsl.executors.status_handling INFO: Allocated block ID 3311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3311 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3311 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3311.1709793947.8723094.submit parsl.executors.status_handling INFO: Allocated block ID 3312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3312 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3312 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3312.1709793947.872738.submit parsl.executors.status_handling INFO: Allocated block ID 3313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3313 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3313 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3313.1709793947.8731668.submit parsl.executors.status_handling INFO: Allocated block ID 3314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3314 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3314 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3314.1709793947.8735857.submit parsl.executors.status_handling INFO: Allocated block ID 3315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3315 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3315 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3315.1709793947.8740137.submit parsl.executors.status_handling INFO: Allocated block ID 3316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3316 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3316 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3316.1709793947.8744447.submit parsl.executors.status_handling INFO: Allocated block ID 3317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3317 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3317 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3317.1709793947.8748715.submit parsl.executors.status_handling INFO: Allocated block ID 3318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3318 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3318 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3318.1709793947.8752973.submit parsl.executors.status_handling INFO: Allocated block ID 3319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3319 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3319 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3319.1709793947.875719.submit parsl.executors.status_handling INFO: Allocated block ID 3320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3320 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3320 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3320.1709793947.8761444.submit parsl.executors.status_handling INFO: Allocated block ID 3321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3321 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3321 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3321.1709793947.8765647.submit parsl.executors.status_handling INFO: Allocated block ID 3322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3322 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3322 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3322.1709793947.876989.submit parsl.executors.status_handling INFO: Allocated block ID 3323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3323 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3323 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3323.1709793947.8774064.submit parsl.executors.status_handling INFO: Allocated block ID 3324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3324 2024-03-06 22:45:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3324 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3324.1709793947.8778408.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a28e356597808919c0927a7a30aba0a7 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/86e78c7d-da31-413a-b18a-b8365217d15f_biasIsr_1123737_47.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/86e78c7d-da31-413a-b18a-b8365217d15f_biasIsr_1123737_47.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abb8e1d2a9fcde4901bdd394a7729e1 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e1067e08-95eb-419f-9646-5965b7287703_biasIsr_1123831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123831/e1067e08-95eb-419f-9646-5965b7287703_biasIsr_1123831_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1348 parsl.dataflow.strategy DEBUG: Executor multi has 1348 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 3325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3325 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3325 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3325.1709793952.654812.submit parsl.executors.status_handling INFO: Allocated block ID 3326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3326 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3326 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3326.1709793952.6554193.submit parsl.executors.status_handling INFO: Allocated block ID 3327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3327 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3327 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3327.1709793952.655984.submit parsl.executors.status_handling INFO: Allocated block ID 3328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3328 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3328 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3328.1709793952.6565223.submit parsl.executors.status_handling INFO: Allocated block ID 3329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3329 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3329 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3329.1709793952.6570668.submit parsl.executors.status_handling INFO: Allocated block ID 3330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3330 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3330 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3330.1709793952.6576018.submit parsl.executors.status_handling INFO: Allocated block ID 3331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3331 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3331 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3331.1709793952.6581497.submit parsl.executors.status_handling INFO: Allocated block ID 3332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3332 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3332 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3332.1709793952.6586835.submit parsl.executors.status_handling INFO: Allocated block ID 3333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3333 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3333 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3333.1709793952.659225.submit parsl.executors.status_handling INFO: Allocated block ID 3334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3334 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3334 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3334.1709793952.6597576.submit parsl.executors.status_handling INFO: Allocated block ID 3335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3335 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3335 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3335.1709793952.6603012.submit parsl.executors.status_handling INFO: Allocated block ID 3336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3336 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3336 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3336.1709793952.660848.submit parsl.executors.status_handling INFO: Allocated block ID 3337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3337 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3337 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3337.1709793952.66139.submit parsl.executors.status_handling INFO: Allocated block ID 3338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3338 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3338 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3338.1709793952.661932.submit parsl.executors.status_handling INFO: Allocated block ID 3339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3339 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3339 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3339.1709793952.6623936.submit parsl.executors.status_handling INFO: Allocated block ID 3340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3340 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3340 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3340.1709793952.6627939.submit parsl.executors.status_handling INFO: Allocated block ID 3341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3341 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3341 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3341.1709793952.6632466.submit parsl.executors.status_handling INFO: Allocated block ID 3342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3342 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3342 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3342.1709793952.6636693.submit parsl.executors.status_handling INFO: Allocated block ID 3343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3343 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3343 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3343.1709793952.6641333.submit parsl.executors.status_handling INFO: Allocated block ID 3344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3344 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3344 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3344.1709793952.664574.submit parsl.executors.status_handling INFO: Allocated block ID 3345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3345 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3345 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3345.1709793952.6650121.submit parsl.executors.status_handling INFO: Allocated block ID 3346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3346 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3346 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3346.1709793952.6654441.submit parsl.executors.status_handling INFO: Allocated block ID 3347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3347 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3347 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3347.1709793952.6658661.submit parsl.executors.status_handling INFO: Allocated block ID 3348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3348 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3348 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3348.1709793952.666308.submit parsl.executors.status_handling INFO: Allocated block ID 3349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3349 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3349 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3349.1709793952.6667318.submit parsl.executors.status_handling INFO: Allocated block ID 3350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3350 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3350 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3350.1709793952.6671593.submit parsl.executors.status_handling INFO: Allocated block ID 3351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3351 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3351 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3351.1709793952.667575.submit parsl.executors.status_handling INFO: Allocated block ID 3352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3352 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3352 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3352.1709793952.6680183.submit parsl.executors.status_handling INFO: Allocated block ID 3353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3353 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3353 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3353.1709793952.6684513.submit parsl.executors.status_handling INFO: Allocated block ID 3354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3354 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3354 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3354.1709793952.6688917.submit parsl.executors.status_handling INFO: Allocated block ID 3355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3355 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3355 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3355.1709793952.669328.submit parsl.executors.status_handling INFO: Allocated block ID 3356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3356 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3356 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3356.1709793952.669744.submit parsl.executors.status_handling INFO: Allocated block ID 3357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3357 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3357 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3357.1709793952.6701725.submit parsl.executors.status_handling INFO: Allocated block ID 3358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3358 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3358 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3358.1709793952.6705909.submit parsl.executors.status_handling INFO: Allocated block ID 3359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3359 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3359 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3359.1709793952.6710162.submit parsl.executors.status_handling INFO: Allocated block ID 3360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3360 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3360 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3360.1709793952.6714406.submit parsl.executors.status_handling INFO: Allocated block ID 3361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3361 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3361 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3361.1709793952.67187.submit parsl.executors.status_handling INFO: Allocated block ID 3362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3362 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3362 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3362.1709793952.6722937.submit parsl.executors.status_handling INFO: Allocated block ID 3363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3363 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3363 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3363.1709793952.6727102.submit parsl.executors.status_handling INFO: Allocated block ID 3364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3364 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3364 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3364.1709793952.6731443.submit parsl.executors.status_handling INFO: Allocated block ID 3365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3365 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3365 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3365.1709793952.6735616.submit parsl.executors.status_handling INFO: Allocated block ID 3366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3366 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3366 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3366.1709793952.673986.submit parsl.executors.status_handling INFO: Allocated block ID 3367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3367 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3367 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3367.1709793952.6744146.submit parsl.executors.status_handling INFO: Allocated block ID 3368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3368 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3368 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3368.1709793952.6748466.submit parsl.executors.status_handling INFO: Allocated block ID 3369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3369 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3369 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3369.1709793952.6752973.submit parsl.executors.status_handling INFO: Allocated block ID 3370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3370 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3370 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3370.1709793952.6757305.submit parsl.executors.status_handling INFO: Allocated block ID 3371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3371 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3371 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3371.1709793952.6761658.submit parsl.executors.status_handling INFO: Allocated block ID 3372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3372 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3372 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3372.1709793952.6766074.submit parsl.executors.status_handling INFO: Allocated block ID 3373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3373 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3373 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3373.1709793952.677052.submit parsl.executors.status_handling INFO: Allocated block ID 3374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3374 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3374 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3374.1709793952.677472.submit parsl.executors.status_handling INFO: Allocated block ID 3375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3375 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3375 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3375.1709793952.6779225.submit parsl.executors.status_handling INFO: Allocated block ID 3376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3376 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3376 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3376.1709793952.6783552.submit parsl.executors.status_handling INFO: Allocated block ID 3377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3377 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3377 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3377.1709793952.6787863.submit parsl.executors.status_handling INFO: Allocated block ID 3378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3378 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3378 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3378.1709793952.6792245.submit parsl.executors.status_handling INFO: Allocated block ID 3379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3379 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3379 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3379.1709793952.6796439.submit parsl.executors.status_handling INFO: Allocated block ID 3380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3380 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3380 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3380.1709793952.6800637.submit parsl.executors.status_handling INFO: Allocated block ID 3381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3381 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3381 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3381.1709793952.6804821.submit parsl.executors.status_handling INFO: Allocated block ID 3382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3382 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3382 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3382.1709793952.6809042.submit parsl.executors.status_handling INFO: Allocated block ID 3383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3383 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3383 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3383.1709793952.681324.submit parsl.executors.status_handling INFO: Allocated block ID 3384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3384 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3384 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3384.1709793952.6817465.submit parsl.executors.status_handling INFO: Allocated block ID 3385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3385 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3385 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3385.1709793952.6821833.submit parsl.executors.status_handling INFO: Allocated block ID 3386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3386 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3386 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3386.1709793952.6826003.submit parsl.executors.status_handling INFO: Allocated block ID 3387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3387 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3387 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3387.1709793952.6830277.submit parsl.executors.status_handling INFO: Allocated block ID 3388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3388 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3388 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3388.1709793952.6834476.submit parsl.executors.status_handling INFO: Allocated block ID 3389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3389 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3389 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3389.1709793952.6838727.submit parsl.executors.status_handling INFO: Allocated block ID 3390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3390 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3390 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3390.1709793952.6842918.submit parsl.executors.status_handling INFO: Allocated block ID 3391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3391 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3391 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3391.1709793952.6847105.submit parsl.executors.status_handling INFO: Allocated block ID 3392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3392 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3392 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3392.1709793952.6851618.submit parsl.executors.status_handling INFO: Allocated block ID 3393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3393 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3393 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3393.1709793952.6855912.submit parsl.executors.status_handling INFO: Allocated block ID 3394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3394 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3394 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3394.1709793952.6860237.submit parsl.executors.status_handling INFO: Allocated block ID 3395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3395 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3395 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3395.1709793952.6864545.submit parsl.executors.status_handling INFO: Allocated block ID 3396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3396 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3396 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3396.1709793952.686886.submit parsl.executors.status_handling INFO: Allocated block ID 3397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3397 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3397 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3397.1709793952.6873374.submit parsl.executors.status_handling INFO: Allocated block ID 3398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3398 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3398 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3398.1709793952.6877646.submit parsl.executors.status_handling INFO: Allocated block ID 3399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3399 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3399 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3399.1709793952.6882062.submit parsl.executors.status_handling INFO: Allocated block ID 3400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3400 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3400 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3400.1709793952.6886213.submit parsl.executors.status_handling INFO: Allocated block ID 3401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3401 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3401 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3401.1709793952.6890507.submit parsl.executors.status_handling INFO: Allocated block ID 3402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3402 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3402 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3402.1709793952.68947.submit parsl.executors.status_handling INFO: Allocated block ID 3403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3403 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3403 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3403.1709793952.6898992.submit parsl.executors.status_handling INFO: Allocated block ID 3404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3404 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3404 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3404.1709793952.690324.submit parsl.executors.status_handling INFO: Allocated block ID 3405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3405 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3405 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3405.1709793952.6907406.submit parsl.executors.status_handling INFO: Allocated block ID 3406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3406 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3406 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3406.1709793952.6911957.submit parsl.executors.status_handling INFO: Allocated block ID 3407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3407 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3407 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3407.1709793952.6916203.submit parsl.executors.status_handling INFO: Allocated block ID 3408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3408 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3408 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3408.1709793952.6920538.submit parsl.executors.status_handling INFO: Allocated block ID 3409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3409 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3409 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3409.1709793952.6924736.submit parsl.executors.status_handling INFO: Allocated block ID 3410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3410 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3410 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3410.1709793952.6929.submit parsl.executors.status_handling INFO: Allocated block ID 3411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3411 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3411 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3411.1709793952.6933208.submit parsl.executors.status_handling INFO: Allocated block ID 3412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3412 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3412 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3412.1709793952.6937387.submit parsl.executors.status_handling INFO: Allocated block ID 3413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3413 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3413 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3413.1709793952.6941645.submit parsl.executors.status_handling INFO: Allocated block ID 3414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3414 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3414 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3414.1709793952.6945825.submit parsl.executors.status_handling INFO: Allocated block ID 3415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3415 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3415 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3415.1709793952.695021.submit parsl.executors.status_handling INFO: Allocated block ID 3416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3416 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3416 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3416.1709793952.6954405.submit parsl.executors.status_handling INFO: Allocated block ID 3417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3417 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3417 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3417.1709793952.6958761.submit parsl.executors.status_handling INFO: Allocated block ID 3418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3418 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3418 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3418.1709793952.6962957.submit parsl.executors.status_handling INFO: Allocated block ID 3419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3419 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3419 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3419.1709793952.6967125.submit parsl.executors.status_handling INFO: Allocated block ID 3420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3420 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3420 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3420.1709793952.697171.submit parsl.executors.status_handling INFO: Allocated block ID 3421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3421 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3421 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3421.1709793952.6976051.submit parsl.executors.status_handling INFO: Allocated block ID 3422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3422 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3422 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3422.1709793952.6980426.submit parsl.executors.status_handling INFO: Allocated block ID 3423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3423 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3423 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3423.1709793952.6984718.submit parsl.executors.status_handling INFO: Allocated block ID 3424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3424 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3424 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3424.1709793952.698894.submit parsl.executors.status_handling INFO: Allocated block ID 3425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3425 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3425 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3425.1709793952.6993127.submit parsl.executors.status_handling INFO: Allocated block ID 3426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3426 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3426 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3426.1709793952.6997514.submit parsl.executors.status_handling INFO: Allocated block ID 3427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3427 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3427 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3427.1709793952.700212.submit parsl.executors.status_handling INFO: Allocated block ID 3428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3428 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3428 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3428.1709793952.7006454.submit parsl.executors.status_handling INFO: Allocated block ID 3429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3429 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3429 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3429.1709793952.701083.submit parsl.executors.status_handling INFO: Allocated block ID 3430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3430 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3430 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3430.1709793952.7015138.submit parsl.executors.status_handling INFO: Allocated block ID 3431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3431 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3431 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3431.1709793952.7019374.submit parsl.executors.status_handling INFO: Allocated block ID 3432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3432 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3432 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3432.1709793952.702365.submit parsl.executors.status_handling INFO: Allocated block ID 3433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3433 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3433 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3433.1709793952.7027876.submit parsl.executors.status_handling INFO: Allocated block ID 3434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3434 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3434 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3434.1709793952.7032156.submit parsl.executors.status_handling INFO: Allocated block ID 3435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3435 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3435 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3435.1709793952.7036343.submit parsl.executors.status_handling INFO: Allocated block ID 3436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3436 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3436 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3436.1709793952.7040734.submit parsl.executors.status_handling INFO: Allocated block ID 3437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3437 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3437 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3437.1709793952.7045028.submit parsl.executors.status_handling INFO: Allocated block ID 3438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3438 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3438 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3438.1709793952.7049391.submit parsl.executors.status_handling INFO: Allocated block ID 3439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3439 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3439 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3439.1709793952.7053704.submit parsl.executors.status_handling INFO: Allocated block ID 3440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3440 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3440 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3440.1709793952.705838.submit parsl.executors.status_handling INFO: Allocated block ID 3441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3441 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3441 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3441.1709793952.706346.submit parsl.executors.status_handling INFO: Allocated block ID 3442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3442 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3442 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3442.1709793952.7067955.submit parsl.executors.status_handling INFO: Allocated block ID 3443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3443 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3443 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3443.1709793952.7072642.submit parsl.executors.status_handling INFO: Allocated block ID 3444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3444 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3444 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3444.1709793952.7077.submit parsl.executors.status_handling INFO: Allocated block ID 3445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3445 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3445 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3445.1709793952.708141.submit parsl.executors.status_handling INFO: Allocated block ID 3446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3446 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3446 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3446.1709793952.7085726.submit parsl.executors.status_handling INFO: Allocated block ID 3447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3447 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3447 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3447.1709793952.7090142.submit parsl.executors.status_handling INFO: Allocated block ID 3448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3448 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3448 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3448.1709793952.709435.submit parsl.executors.status_handling INFO: Allocated block ID 3449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3449 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3449 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3449.1709793952.7098603.submit parsl.executors.status_handling INFO: Allocated block ID 3450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3450 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3450 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3450.1709793952.7102935.submit parsl.executors.status_handling INFO: Allocated block ID 3451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3451 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3451 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3451.1709793952.7107248.submit parsl.executors.status_handling INFO: Allocated block ID 3452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3452 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3452 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3452.1709793952.711159.submit parsl.executors.status_handling INFO: Allocated block ID 3453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3453 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3453 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3453.1709793952.7115912.submit parsl.executors.status_handling INFO: Allocated block ID 3454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3454 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3454 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3454.1709793952.712025.submit parsl.executors.status_handling INFO: Allocated block ID 3455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3455 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3455 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3455.1709793952.7124517.submit parsl.executors.status_handling INFO: Allocated block ID 3456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3456 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3456 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3456.1709793952.712896.submit parsl.executors.status_handling INFO: Allocated block ID 3457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3457 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3457 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3457.1709793952.713329.submit parsl.executors.status_handling INFO: Allocated block ID 3458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3458 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3458 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3458.1709793952.713742.submit parsl.executors.status_handling INFO: Allocated block ID 3459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3459 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3459 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3459.1709793952.7141645.submit parsl.executors.status_handling INFO: Allocated block ID 3460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3460 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3460 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3460.1709793952.7145772.submit parsl.executors.status_handling INFO: Allocated block ID 3461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3461 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3461 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3461.1709793952.7150307.submit parsl.executors.status_handling INFO: Allocated block ID 3462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3462 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3462 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3462.1709793952.7154832.submit parsl.executors.status_handling INFO: Allocated block ID 3463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3463 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3463 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3463.1709793952.7159498.submit parsl.executors.status_handling INFO: Allocated block ID 3464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3464 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3464 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3464.1709793952.7164078.submit parsl.executors.status_handling INFO: Allocated block ID 3465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3465 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3465 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3465.1709793952.7168438.submit parsl.executors.status_handling INFO: Allocated block ID 3466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3466 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3466 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3466.1709793952.7172744.submit parsl.executors.status_handling INFO: Allocated block ID 3467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3467 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3467 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3467.1709793952.7177134.submit parsl.executors.status_handling INFO: Allocated block ID 3468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3468 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3468 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3468.1709793952.7181435.submit parsl.executors.status_handling INFO: Allocated block ID 3469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3469 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3469 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3469.1709793952.7185647.submit parsl.executors.status_handling INFO: Allocated block ID 3470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3470 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3470 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3470.1709793952.7189946.submit parsl.executors.status_handling INFO: Allocated block ID 3471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3471 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3471 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3471.1709793952.7194302.submit parsl.executors.status_handling INFO: Allocated block ID 3472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3472 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3472 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3472.1709793952.719885.submit parsl.executors.status_handling INFO: Allocated block ID 3473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3473 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3473 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3473.1709793952.7203429.submit parsl.executors.status_handling INFO: Allocated block ID 3474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3474 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3474 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3474.1709793952.7208047.submit parsl.executors.status_handling INFO: Allocated block ID 3475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3475 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3475 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3475.1709793952.7212715.submit parsl.executors.status_handling INFO: Allocated block ID 3476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3476 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3476 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3476.1709793952.721722.submit parsl.executors.status_handling INFO: Allocated block ID 3477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3477 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3477 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3477.1709793952.7221622.submit parsl.executors.status_handling INFO: Allocated block ID 3478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3478 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3478 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3478.1709793952.7225916.submit parsl.executors.status_handling INFO: Allocated block ID 3479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3479 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3479 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3479.1709793952.7230246.submit parsl.executors.status_handling INFO: Allocated block ID 3480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3480 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3480 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3480.1709793952.723461.submit parsl.executors.status_handling INFO: Allocated block ID 3481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3481 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3481 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3481.1709793952.723889.submit parsl.executors.status_handling INFO: Allocated block ID 3482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3482 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3482 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3482.1709793952.7243078.submit parsl.executors.status_handling INFO: Allocated block ID 3483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3483 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3483 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3483.1709793952.7248864.submit parsl.executors.status_handling INFO: Allocated block ID 3484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3484 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3484 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3484.1709793952.7253153.submit parsl.executors.status_handling INFO: Allocated block ID 3485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3485 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3485 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3485.1709793952.7257526.submit parsl.executors.status_handling INFO: Allocated block ID 3486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3486 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3486 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3486.1709793952.7261908.submit parsl.executors.status_handling INFO: Allocated block ID 3487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3487 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3487 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3487.1709793952.726625.submit parsl.executors.status_handling INFO: Allocated block ID 3488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3488 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3488 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3488.1709793952.7270596.submit parsl.executors.status_handling INFO: Allocated block ID 3489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3489 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3489 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3489.1709793952.727489.submit parsl.executors.status_handling INFO: Allocated block ID 3490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3490 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3490 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3490.1709793952.7279108.submit parsl.executors.status_handling INFO: Allocated block ID 3491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3491 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3491 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3491.1709793952.7283309.submit parsl.executors.status_handling INFO: Allocated block ID 3492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3492 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3492 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3492.1709793952.7287855.submit parsl.executors.status_handling INFO: Allocated block ID 3493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3493 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3493 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3493.1709793952.7292304.submit parsl.executors.status_handling INFO: Allocated block ID 3494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3494 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3494 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3494.1709793952.729661.submit parsl.executors.status_handling INFO: Allocated block ID 3495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3495 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3495 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3495.1709793952.7301013.submit parsl.executors.status_handling INFO: Allocated block ID 3496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3496 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3496 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3496.1709793952.7305346.submit parsl.executors.status_handling INFO: Allocated block ID 3497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3497 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3497 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3497.1709793952.7309816.submit parsl.executors.status_handling INFO: Allocated block ID 3498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3498 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3498 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3498.1709793952.7314005.submit parsl.executors.status_handling INFO: Allocated block ID 3499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3499 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3499 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3499.1709793952.7318544.submit parsl.executors.status_handling INFO: Allocated block ID 3500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3500 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3500 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3500.1709793952.73228.submit parsl.executors.status_handling INFO: Allocated block ID 3501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3501 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3501 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3501.1709793952.732716.submit parsl.executors.status_handling INFO: Allocated block ID 3502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3502 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3502 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3502.1709793952.733138.submit parsl.executors.status_handling INFO: Allocated block ID 3503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3503 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3503 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3503.1709793952.7335513.submit parsl.executors.status_handling INFO: Allocated block ID 3504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3504 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3504 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3504.1709793952.7339838.submit parsl.executors.status_handling INFO: Allocated block ID 3505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3505 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3505 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3505.1709793952.7344081.submit parsl.executors.status_handling INFO: Allocated block ID 3506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3506 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3506 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3506.1709793952.7348323.submit parsl.executors.status_handling INFO: Allocated block ID 3507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3507 2024-03-06 22:45:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3507 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3507.1709793952.735258.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1348 parsl.dataflow.strategy DEBUG: Executor multi has 1348 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 3508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3508 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3508 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3508.1709793957.6551054.submit parsl.executors.status_handling INFO: Allocated block ID 3509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3509 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3509 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3509.1709793957.655598.submit parsl.executors.status_handling INFO: Allocated block ID 3510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3510 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3510 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3510.1709793957.656058.submit parsl.executors.status_handling INFO: Allocated block ID 3511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3511 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3511 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3511.1709793957.6564934.submit parsl.executors.status_handling INFO: Allocated block ID 3512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3512 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3512 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3512.1709793957.6569312.submit parsl.executors.status_handling INFO: Allocated block ID 3513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3513 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3513 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3513.1709793957.6573744.submit parsl.executors.status_handling INFO: Allocated block ID 3514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3514 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3514 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3514.1709793957.6578035.submit parsl.executors.status_handling INFO: Allocated block ID 3515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3515 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3515 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3515.1709793957.6582415.submit parsl.executors.status_handling INFO: Allocated block ID 3516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3516 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3516 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3516.1709793957.6586637.submit parsl.executors.status_handling INFO: Allocated block ID 3517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3517 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3517 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3517.1709793957.6590965.submit parsl.executors.status_handling INFO: Allocated block ID 3518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3518 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3518 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3518.1709793957.6595197.submit parsl.executors.status_handling INFO: Allocated block ID 3519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3519 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3519 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3519.1709793957.6599643.submit parsl.executors.status_handling INFO: Allocated block ID 3520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3520 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3520 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3520.1709793957.6603975.submit parsl.executors.status_handling INFO: Allocated block ID 3521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3521 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3521 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3521.1709793957.6608424.submit parsl.executors.status_handling INFO: Allocated block ID 3522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3522 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3522 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3522.1709793957.661278.submit parsl.executors.status_handling INFO: Allocated block ID 3523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3523 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3523 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3523.1709793957.661715.submit parsl.executors.status_handling INFO: Allocated block ID 3524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3524 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3524 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3524.1709793957.662159.submit parsl.executors.status_handling INFO: Allocated block ID 3525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3525 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3525 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3525.1709793957.6625757.submit parsl.executors.status_handling INFO: Allocated block ID 3526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3526 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3526 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3526.1709793957.6630101.submit parsl.executors.status_handling INFO: Allocated block ID 3527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3527 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3527 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3527.1709793957.663436.submit parsl.executors.status_handling INFO: Allocated block ID 3528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3528 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3528 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3528.1709793957.6638591.submit parsl.executors.status_handling INFO: Allocated block ID 3529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3529 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3529 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3529.1709793957.6642797.submit parsl.executors.status_handling INFO: Allocated block ID 3530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3530 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3530 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3530.1709793957.6646988.submit parsl.executors.status_handling INFO: Allocated block ID 3531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3531 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3531 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3531.1709793957.66513.submit parsl.executors.status_handling INFO: Allocated block ID 3532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3532 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3532 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3532.1709793957.6655514.submit parsl.executors.status_handling INFO: Allocated block ID 3533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3533 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3533 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3533.1709793957.6659856.submit parsl.executors.status_handling INFO: Allocated block ID 3534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3534 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3534 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3534.1709793957.6664112.submit parsl.executors.status_handling INFO: Allocated block ID 3535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3535 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3535 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3535.1709793957.6799395.submit parsl.executors.status_handling INFO: Allocated block ID 3536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3536 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3536 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3536.1709793957.6804545.submit parsl.executors.status_handling INFO: Allocated block ID 3537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3537 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3537 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3537.1709793957.6808972.submit parsl.executors.status_handling INFO: Allocated block ID 3538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3538 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3538 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3538.1709793957.6813316.submit parsl.executors.status_handling INFO: Allocated block ID 3539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3539 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3539 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3539.1709793957.6817605.submit parsl.executors.status_handling INFO: Allocated block ID 3540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3540 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3540 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3540.1709793957.6821923.submit parsl.executors.status_handling INFO: Allocated block ID 3541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3541 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3541 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3541.1709793957.6826158.submit parsl.executors.status_handling INFO: Allocated block ID 3542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3542 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3542 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3542.1709793957.6830413.submit parsl.executors.status_handling INFO: Allocated block ID 3543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3543 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3543 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3543.1709793957.6834643.submit parsl.executors.status_handling INFO: Allocated block ID 3544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3544 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3544 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3544.1709793957.6838913.submit parsl.executors.status_handling INFO: Allocated block ID 3545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3545 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3545 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3545.1709793957.684314.submit parsl.executors.status_handling INFO: Allocated block ID 3546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3546 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3546 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3546.1709793957.684733.submit parsl.executors.status_handling INFO: Allocated block ID 3547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3547 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3547 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3547.1709793957.6851852.submit parsl.executors.status_handling INFO: Allocated block ID 3548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3548 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3548 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3548.1709793957.6856165.submit parsl.executors.status_handling INFO: Allocated block ID 3549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3549 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3549 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3549.1709793957.686056.submit parsl.executors.status_handling INFO: Allocated block ID 3550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3550 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3550 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3550.1709793957.6864917.submit parsl.executors.status_handling INFO: Allocated block ID 3551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3551 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3551 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3551.1709793957.6869254.submit parsl.executors.status_handling INFO: Allocated block ID 3552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3552 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3552 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3552.1709793957.6873507.submit parsl.executors.status_handling INFO: Allocated block ID 3553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3553 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3553 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3553.1709793957.6877706.submit parsl.executors.status_handling INFO: Allocated block ID 3554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3554 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3554 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3554.1709793957.6882226.submit parsl.executors.status_handling INFO: Allocated block ID 3555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3555 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3555 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3555.1709793957.6886594.submit parsl.executors.status_handling INFO: Allocated block ID 3556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3556 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3556 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3556.1709793957.6891072.submit parsl.executors.status_handling INFO: Allocated block ID 3557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3557 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3557 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3557.1709793957.6895308.submit parsl.executors.status_handling INFO: Allocated block ID 3558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3558 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3558 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3558.1709793957.689974.submit parsl.executors.status_handling INFO: Allocated block ID 3559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3559 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3559 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3559.1709793957.6904066.submit parsl.executors.status_handling INFO: Allocated block ID 3560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3560 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3560 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3560.1709793957.69084.submit parsl.executors.status_handling INFO: Allocated block ID 3561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3561 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3561 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3561.1709793957.6912718.submit parsl.executors.status_handling INFO: Allocated block ID 3562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3562 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3562 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3562.1709793957.691709.submit parsl.executors.status_handling INFO: Allocated block ID 3563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3563 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3563 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3563.1709793957.6921525.submit parsl.executors.status_handling INFO: Allocated block ID 3564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3564 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3564 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3564.1709793957.6925845.submit parsl.executors.status_handling INFO: Allocated block ID 3565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3565 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3565 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3565.1709793957.706127.submit parsl.executors.status_handling INFO: Allocated block ID 3566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3566 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3566 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3566.1709793957.7066166.submit parsl.executors.status_handling INFO: Allocated block ID 3567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3567 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3567 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3567.1709793957.7070632.submit parsl.executors.status_handling INFO: Allocated block ID 3568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3568 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3568 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3568.1709793957.7074869.submit parsl.executors.status_handling INFO: Allocated block ID 3569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3569 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3569 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3569.1709793957.7079365.submit parsl.executors.status_handling INFO: Allocated block ID 3570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3570 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3570 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3570.1709793957.7083666.submit parsl.executors.status_handling INFO: Allocated block ID 3571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3571 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3571 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3571.1709793957.708793.submit parsl.executors.status_handling INFO: Allocated block ID 3572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3572 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3572 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3572.1709793957.7092388.submit parsl.executors.status_handling INFO: Allocated block ID 3573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3573 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3573 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3573.1709793957.7096577.submit parsl.executors.status_handling INFO: Allocated block ID 3574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3574 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3574 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3574.1709793957.7100904.submit parsl.executors.status_handling INFO: Allocated block ID 3575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3575 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3575 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3575.1709793957.7105153.submit parsl.executors.status_handling INFO: Allocated block ID 3576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3576 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3576 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3576.1709793957.7109392.submit parsl.executors.status_handling INFO: Allocated block ID 3577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3577 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3577 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3577.1709793957.7113693.submit parsl.executors.status_handling INFO: Allocated block ID 3578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3578 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3578 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3578.1709793957.711795.submit parsl.executors.status_handling INFO: Allocated block ID 3579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3579 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3579 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3579.1709793957.7122295.submit parsl.executors.status_handling INFO: Allocated block ID 3580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3580 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3580 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3580.1709793957.7126517.submit parsl.executors.status_handling INFO: Allocated block ID 3581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3581 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3581 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3581.1709793957.7130823.submit parsl.executors.status_handling INFO: Allocated block ID 3582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3582 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3582 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3582.1709793957.7135043.submit parsl.executors.status_handling INFO: Allocated block ID 3583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3583 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3583 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3583.1709793957.713936.submit parsl.executors.status_handling INFO: Allocated block ID 3584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3584 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3584 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3584.1709793957.7143493.submit parsl.executors.status_handling INFO: Allocated block ID 3585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3585 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3585 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3585.1709793957.71477.submit parsl.executors.status_handling INFO: Allocated block ID 3586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3586 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3586 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3586.1709793957.7152042.submit parsl.executors.status_handling INFO: Allocated block ID 3587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3587 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3587 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3587.1709793957.7156305.submit parsl.executors.status_handling INFO: Allocated block ID 3588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3588 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3588 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3588.1709793957.716057.submit parsl.executors.status_handling INFO: Allocated block ID 3589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3589 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3589 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3589.1709793957.7164757.submit parsl.executors.status_handling INFO: Allocated block ID 3590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3590 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3590 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3590.1709793957.7168999.submit parsl.executors.status_handling INFO: Allocated block ID 3591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3591 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3591 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3591.1709793957.7173207.submit parsl.executors.status_handling INFO: Allocated block ID 3592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3592 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3592 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3592.1709793957.7177396.submit parsl.executors.status_handling INFO: Allocated block ID 3593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3593 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3593 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3593.1709793957.7181697.submit parsl.executors.status_handling INFO: Allocated block ID 3594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3594 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3594 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3594.1709793957.7185879.submit parsl.executors.status_handling INFO: Allocated block ID 3595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3595 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3595 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3595.1709793957.7321522.submit parsl.executors.status_handling INFO: Allocated block ID 3596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3596 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3596 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3596.1709793957.7326562.submit parsl.executors.status_handling INFO: Allocated block ID 3597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3597 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3597 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3597.1709793957.73311.submit parsl.executors.status_handling INFO: Allocated block ID 3598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3598 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3598 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3598.1709793957.7335577.submit parsl.executors.status_handling INFO: Allocated block ID 3599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3599 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3599 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3599.1709793957.7340086.submit parsl.executors.status_handling INFO: Allocated block ID 3600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3600 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3600 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3600.1709793957.734434.submit parsl.executors.status_handling INFO: Allocated block ID 3601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3601 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3601 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3601.1709793957.7348597.submit parsl.executors.status_handling INFO: Allocated block ID 3602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3602 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3602 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3602.1709793957.7352808.submit parsl.executors.status_handling INFO: Allocated block ID 3603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3603 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3603 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3603.1709793957.7357023.submit parsl.executors.status_handling INFO: Allocated block ID 3604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3604 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3604 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3604.1709793957.736154.submit parsl.executors.status_handling INFO: Allocated block ID 3605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3605 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3605 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3605.1709793957.7365882.submit parsl.executors.status_handling INFO: Allocated block ID 3606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3606 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3606 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3606.1709793957.737026.submit parsl.executors.status_handling INFO: Allocated block ID 3607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3607 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3607 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3607.1709793957.737456.submit parsl.executors.status_handling INFO: Allocated block ID 3608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3608 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3608 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3608.1709793957.737881.submit parsl.executors.status_handling INFO: Allocated block ID 3609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3609 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3609 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3609.1709793957.7383015.submit parsl.executors.status_handling INFO: Allocated block ID 3610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3610 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3610 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3610.1709793957.738727.submit parsl.executors.status_handling INFO: Allocated block ID 3611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3611 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3611 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3611.1709793957.7391582.submit parsl.executors.status_handling INFO: Allocated block ID 3612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3612 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3612 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3612.1709793957.739586.submit parsl.executors.status_handling INFO: Allocated block ID 3613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3613 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3613 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3613.1709793957.7400148.submit parsl.executors.status_handling INFO: Allocated block ID 3614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3614 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3614 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3614.1709793957.7404363.submit parsl.executors.status_handling INFO: Allocated block ID 3615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3615 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3615 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3615.1709793957.7408779.submit parsl.executors.status_handling INFO: Allocated block ID 3616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3616 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3616 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3616.1709793957.7413082.submit parsl.executors.status_handling INFO: Allocated block ID 3617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3617 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3617 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3617.1709793957.741741.submit parsl.executors.status_handling INFO: Allocated block ID 3618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3618 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3618 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3618.1709793957.7421865.submit parsl.executors.status_handling INFO: Allocated block ID 3619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3619 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3619 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3619.1709793957.7426114.submit parsl.executors.status_handling INFO: Allocated block ID 3620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3620 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3620 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3620.1709793957.7430363.submit parsl.executors.status_handling INFO: Allocated block ID 3621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3621 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3621 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3621.1709793957.7434742.submit parsl.executors.status_handling INFO: Allocated block ID 3622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3622 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3622 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3622.1709793957.7439187.submit parsl.executors.status_handling INFO: Allocated block ID 3623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3623 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3623 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3623.1709793957.7443547.submit parsl.executors.status_handling INFO: Allocated block ID 3624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3624 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3624 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3624.1709793957.7447948.submit parsl.executors.status_handling INFO: Allocated block ID 3625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3625 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3625 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3625.1709793957.7583828.submit parsl.executors.status_handling INFO: Allocated block ID 3626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3626 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3626 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3626.1709793957.758814.submit parsl.executors.status_handling INFO: Allocated block ID 3627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3627 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3627 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3627.1709793957.7592463.submit parsl.executors.status_handling INFO: Allocated block ID 3628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3628 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3628 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3628.1709793957.7596664.submit parsl.executors.status_handling INFO: Allocated block ID 3629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3629 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3629 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3629.1709793957.7600946.submit parsl.executors.status_handling INFO: Allocated block ID 3630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3630 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3630 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3630.1709793957.760518.submit parsl.executors.status_handling INFO: Allocated block ID 3631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3631 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3631 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3631.1709793957.760945.submit parsl.executors.status_handling INFO: Allocated block ID 3632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3632 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3632 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3632.1709793957.7613623.submit parsl.executors.status_handling INFO: Allocated block ID 3633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3633 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3633 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3633.1709793957.7617788.submit parsl.executors.status_handling INFO: Allocated block ID 3634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3634 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3634 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3634.1709793957.7622123.submit parsl.executors.status_handling INFO: Allocated block ID 3635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3635 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3635 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3635.1709793957.7626374.submit parsl.executors.status_handling INFO: Allocated block ID 3636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3636 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3636 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3636.1709793957.7630658.submit parsl.executors.status_handling INFO: Allocated block ID 3637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3637 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3637 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3637.1709793957.7634847.submit parsl.executors.status_handling INFO: Allocated block ID 3638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3638 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3638 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3638.1709793957.7639112.submit parsl.executors.status_handling INFO: Allocated block ID 3639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3639 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3639 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3639.1709793957.7643313.submit parsl.executors.status_handling INFO: Allocated block ID 3640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3640 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3640 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3640.1709793957.7647588.submit parsl.executors.status_handling INFO: Allocated block ID 3641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3641 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3641 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3641.1709793957.7652123.submit parsl.executors.status_handling INFO: Allocated block ID 3642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3642 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3642 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3642.1709793957.7656424.submit parsl.executors.status_handling INFO: Allocated block ID 3643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3643 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3643 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3643.1709793957.7661054.submit parsl.executors.status_handling INFO: Allocated block ID 3644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3644 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3644 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3644.1709793957.7665267.submit parsl.executors.status_handling INFO: Allocated block ID 3645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3645 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3645 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3645.1709793957.766952.submit parsl.executors.status_handling INFO: Allocated block ID 3646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3646 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3646 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3646.1709793957.7673776.submit parsl.executors.status_handling INFO: Allocated block ID 3647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3647 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3647 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3647.1709793957.7678044.submit parsl.executors.status_handling INFO: Allocated block ID 3648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3648 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3648 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3648.1709793957.768256.submit parsl.executors.status_handling INFO: Allocated block ID 3649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3649 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3649 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3649.1709793957.768688.submit parsl.executors.status_handling INFO: Allocated block ID 3650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3650 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3650 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3650.1709793957.769133.submit parsl.executors.status_handling INFO: Allocated block ID 3651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3651 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3651 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3651.1709793957.769558.submit parsl.executors.status_handling INFO: Allocated block ID 3652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3652 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3652 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3652.1709793957.7699857.submit parsl.executors.status_handling INFO: Allocated block ID 3653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3653 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3653 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3653.1709793957.770411.submit parsl.executors.status_handling INFO: Allocated block ID 3654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3654 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3654 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3654.1709793957.783906.submit parsl.executors.status_handling INFO: Allocated block ID 3655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3655 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3655 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3655.1709793957.7844021.submit parsl.executors.status_handling INFO: Allocated block ID 3656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3656 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3656 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3656.1709793957.7848418.submit parsl.executors.status_handling INFO: Allocated block ID 3657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3657 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3657 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3657.1709793957.7852738.submit parsl.executors.status_handling INFO: Allocated block ID 3658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3658 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3658 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3658.1709793957.785703.submit parsl.executors.status_handling INFO: Allocated block ID 3659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3659 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3659 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3659.1709793957.7861385.submit parsl.executors.status_handling INFO: Allocated block ID 3660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3660 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3660 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3660.1709793957.7865648.submit parsl.executors.status_handling INFO: Allocated block ID 3661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3661 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3661 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3661.1709793957.7869945.submit parsl.executors.status_handling INFO: Allocated block ID 3662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3662 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3662 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3662.1709793957.7874246.submit parsl.executors.status_handling INFO: Allocated block ID 3663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3663 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3663 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3663.1709793957.7878728.submit parsl.executors.status_handling INFO: Allocated block ID 3664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3664 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3664 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3664.1709793957.788308.submit parsl.executors.status_handling INFO: Allocated block ID 3665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3665 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3665 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3665.1709793957.7887444.submit parsl.executors.status_handling INFO: Allocated block ID 3666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3666 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3666 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3666.1709793957.7891843.submit parsl.executors.status_handling INFO: Allocated block ID 3667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3667 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3667 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3667.1709793957.7896104.submit parsl.executors.status_handling INFO: Allocated block ID 3668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3668 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3668 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3668.1709793957.7900562.submit parsl.executors.status_handling INFO: Allocated block ID 3669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3669 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3669 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3669.1709793957.7904758.submit parsl.executors.status_handling INFO: Allocated block ID 3670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3670 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3670 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3670.1709793957.7909281.submit parsl.executors.status_handling INFO: Allocated block ID 3671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3671 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3671 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3671.1709793957.7913663.submit parsl.executors.status_handling INFO: Allocated block ID 3672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3672 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3672 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3672.1709793957.7918038.submit parsl.executors.status_handling INFO: Allocated block ID 3673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3673 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3673 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3673.1709793957.7922451.submit parsl.executors.status_handling INFO: Allocated block ID 3674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3674 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3674 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3674.1709793957.7926776.submit parsl.executors.status_handling INFO: Allocated block ID 3675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3675 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3675 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3675.1709793957.7931159.submit parsl.executors.status_handling INFO: Allocated block ID 3676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3676 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3676 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3676.1709793957.7935498.submit parsl.executors.status_handling INFO: Allocated block ID 3677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3677 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3677 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3677.1709793957.7940047.submit parsl.executors.status_handling INFO: Allocated block ID 3678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3678 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3678 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3678.1709793957.7944243.submit parsl.executors.status_handling INFO: Allocated block ID 3679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3679 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3679 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3679.1709793957.7948654.submit parsl.executors.status_handling INFO: Allocated block ID 3680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3680 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3680 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3680.1709793957.795298.submit parsl.executors.status_handling INFO: Allocated block ID 3681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3681 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3681 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3681.1709793957.7957282.submit parsl.executors.status_handling INFO: Allocated block ID 3682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3682 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3682 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3682.1709793957.7961736.submit parsl.executors.status_handling INFO: Allocated block ID 3683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3683 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3683 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3683.1709793957.796619.submit parsl.executors.status_handling INFO: Allocated block ID 3684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3684 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3684 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3684.1709793957.8102305.submit parsl.executors.status_handling INFO: Allocated block ID 3685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3685 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3685 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3685.1709793957.8107085.submit parsl.executors.status_handling INFO: Allocated block ID 3686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3686 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3686 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3686.1709793957.8111475.submit parsl.executors.status_handling INFO: Allocated block ID 3687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3687 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3687 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3687.1709793957.8115792.submit parsl.executors.status_handling INFO: Allocated block ID 3688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3688 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3688 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3688.1709793957.8120286.submit parsl.executors.status_handling INFO: Allocated block ID 3689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3689 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3689 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3689.1709793957.8124654.submit parsl.executors.status_handling INFO: Allocated block ID 3690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3690 2024-03-06 22:45:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3690 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:45:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:45:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:45:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:45:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3690.1709793957.8129022.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7f625f010b8a5de97d56ca55db47d58 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/bd944be8-8265-4fb3-9059-2240585f6e0a_biasIsr_1123829_47.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123829/bd944be8-8265-4fb3-9059-2240585f6e0a_biasIsr_1123829_47.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2d9f4441d4ab0fe5c0fa83f207265b with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/081691c3-3973-44c5-9b13-25852248af7b_biasIsr_1123735_47.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/081691c3-3973-44c5-9b13-25852248af7b_biasIsr_1123735_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1346 parsl.dataflow.strategy DEBUG: Executor multi has 1346 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 3691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3691 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3691 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3691.1709793962.6654432.submit parsl.executors.status_handling INFO: Allocated block ID 3692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3692 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3692 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3692.1709793962.6659305.submit parsl.executors.status_handling INFO: Allocated block ID 3693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3693 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3693 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3693.1709793962.6663702.submit parsl.executors.status_handling INFO: Allocated block ID 3694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3694 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3694 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3694.1709793962.6667974.submit parsl.executors.status_handling INFO: Allocated block ID 3695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3695 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3695 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3695.1709793962.6672375.submit parsl.executors.status_handling INFO: Allocated block ID 3696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3696 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3696 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3696.1709793962.6676621.submit parsl.executors.status_handling INFO: Allocated block ID 3697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3697 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3697 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3697.1709793962.6681032.submit parsl.executors.status_handling INFO: Allocated block ID 3698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3698 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3698 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3698.1709793962.668528.submit parsl.executors.status_handling INFO: Allocated block ID 3699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3699 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3699 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3699.1709793962.6689587.submit parsl.executors.status_handling INFO: Allocated block ID 3700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3700 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3700 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3700.1709793962.6693854.submit parsl.executors.status_handling INFO: Allocated block ID 3701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3701 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3701 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3701.1709793962.6698127.submit parsl.executors.status_handling INFO: Allocated block ID 3702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3702 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3702 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3702.1709793962.670248.submit parsl.executors.status_handling INFO: Allocated block ID 3703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3703 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3703 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3703.1709793962.6706872.submit parsl.executors.status_handling INFO: Allocated block ID 3704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3704 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3704 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3704.1709793962.6711273.submit parsl.executors.status_handling INFO: Allocated block ID 3705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3705 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3705 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3705.1709793962.671548.submit parsl.executors.status_handling INFO: Allocated block ID 3706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3706 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3706 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3706.1709793962.6719804.submit parsl.executors.status_handling INFO: Allocated block ID 3707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3707 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3707 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3707.1709793962.6724098.submit parsl.executors.status_handling INFO: Allocated block ID 3708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3708 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3708 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3708.1709793962.672835.submit parsl.executors.status_handling INFO: Allocated block ID 3709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3709 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3709 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3709.1709793962.6732614.submit parsl.executors.status_handling INFO: Allocated block ID 3710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3710 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3710 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3710.1709793962.673682.submit parsl.executors.status_handling INFO: Allocated block ID 3711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3711 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3711 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3711.1709793962.6741178.submit parsl.executors.status_handling INFO: Allocated block ID 3712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3712 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3712 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3712.1709793962.6745455.submit parsl.executors.status_handling INFO: Allocated block ID 3713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3713 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3713 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3713.1709793962.6840882.submit parsl.executors.status_handling INFO: Allocated block ID 3714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3714 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3714 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3714.1709793962.6845727.submit parsl.executors.status_handling INFO: Allocated block ID 3715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3715 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3715 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3715.1709793962.685043.submit parsl.executors.status_handling INFO: Allocated block ID 3716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3716 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3716 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3716.1709793962.685488.submit parsl.executors.status_handling INFO: Allocated block ID 3717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3717 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3717 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3717.1709793962.68593.submit parsl.executors.status_handling INFO: Allocated block ID 3718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3718 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3718 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3718.1709793962.6863544.submit parsl.executors.status_handling INFO: Allocated block ID 3719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3719 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3719 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3719.1709793962.686782.submit parsl.executors.status_handling INFO: Allocated block ID 3720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3720 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3720 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3720.1709793962.6872084.submit parsl.executors.status_handling INFO: Allocated block ID 3721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3721 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3721 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3721.1709793962.6876376.submit parsl.executors.status_handling INFO: Allocated block ID 3722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3722 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3722 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3722.1709793962.6880736.submit parsl.executors.status_handling INFO: Allocated block ID 3723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3723 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3723 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3723.1709793962.688499.submit parsl.executors.status_handling INFO: Allocated block ID 3724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3724 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3724 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3724.1709793962.688927.submit parsl.executors.status_handling INFO: Allocated block ID 3725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3725 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3725 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3725.1709793962.6893601.submit parsl.executors.status_handling INFO: Allocated block ID 3726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3726 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3726 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3726.1709793962.6897933.submit parsl.executors.status_handling INFO: Allocated block ID 3727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3727 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3727 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3727.1709793962.6902313.submit parsl.executors.status_handling INFO: Allocated block ID 3728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3728 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3728 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3728.1709793962.6906579.submit parsl.executors.status_handling INFO: Allocated block ID 3729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3729 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3729 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3729.1709793962.6910872.submit parsl.executors.status_handling INFO: Allocated block ID 3730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3730 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3730 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3730.1709793962.6915166.submit parsl.executors.status_handling INFO: Allocated block ID 3731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3731 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3731 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3731.1709793962.691954.submit parsl.executors.status_handling INFO: Allocated block ID 3732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3732 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3732 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3732.1709793962.692388.submit parsl.executors.status_handling INFO: Allocated block ID 3733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3733 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3733 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3733.1709793962.6928272.submit parsl.executors.status_handling INFO: Allocated block ID 3734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3734 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3734 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3734.1709793962.6932492.submit parsl.executors.status_handling INFO: Allocated block ID 3735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3735 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3735 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3735.1709793962.6936746.submit parsl.executors.status_handling INFO: Allocated block ID 3736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3736 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3736 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3736.1709793962.6941051.submit parsl.executors.status_handling INFO: Allocated block ID 3737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3737 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3737 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3737.1709793962.694533.submit parsl.executors.status_handling INFO: Allocated block ID 3738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3738 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3738 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3738.1709793962.6949685.submit parsl.executors.status_handling INFO: Allocated block ID 3739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3739 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3739 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3739.1709793962.6954038.submit parsl.executors.status_handling INFO: Allocated block ID 3740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3740 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3740 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3740.1709793962.6958282.submit parsl.executors.status_handling INFO: Allocated block ID 3741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3741 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3741 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3741.1709793962.6962528.submit parsl.executors.status_handling INFO: Allocated block ID 3742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3742 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3742 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3742.1709793962.6966758.submit parsl.executors.status_handling INFO: Allocated block ID 3743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3743 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3743 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3743.1709793962.697106.submit parsl.executors.status_handling INFO: Allocated block ID 3744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3744 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3744 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3744.1709793962.697526.submit parsl.executors.status_handling INFO: Allocated block ID 3745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3745 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3745 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3745.1709793962.6979585.submit parsl.executors.status_handling INFO: Allocated block ID 3746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3746 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3746 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3746.1709793962.6984289.submit parsl.executors.status_handling INFO: Allocated block ID 3747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3747 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3747 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3747.1709793962.7119572.submit parsl.executors.status_handling INFO: Allocated block ID 3748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3748 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3748 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3748.1709793962.712436.submit parsl.executors.status_handling INFO: Allocated block ID 3749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3749 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3749 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3749.1709793962.712881.submit parsl.executors.status_handling INFO: Allocated block ID 3750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3750 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3750 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3750.1709793962.713319.submit parsl.executors.status_handling INFO: Allocated block ID 3751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3751 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3751 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3751.1709793962.713752.submit parsl.executors.status_handling INFO: Allocated block ID 3752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3752 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3752 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3752.1709793962.7141893.submit parsl.executors.status_handling INFO: Allocated block ID 3753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3753 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3753 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3753.1709793962.7146153.submit parsl.executors.status_handling INFO: Allocated block ID 3754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3754 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3754 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3754.1709793962.7150488.submit parsl.executors.status_handling INFO: Allocated block ID 3755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3755 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3755 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3755.1709793962.7154722.submit parsl.executors.status_handling INFO: Allocated block ID 3756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3756 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3756 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3756.1709793962.7158995.submit parsl.executors.status_handling INFO: Allocated block ID 3757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3757 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3757 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3757.1709793962.7163265.submit parsl.executors.status_handling INFO: Allocated block ID 3758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3758 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3758 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3758.1709793962.7167506.submit parsl.executors.status_handling INFO: Allocated block ID 3759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3759 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3759 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3759.1709793962.717182.submit parsl.executors.status_handling INFO: Allocated block ID 3760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3760 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3760 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3760.1709793962.7176108.submit parsl.executors.status_handling INFO: Allocated block ID 3761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3761 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3761 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3761.1709793962.7303185.submit parsl.executors.status_handling INFO: Allocated block ID 3762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3762 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3762 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3762.1709793962.7307584.submit parsl.executors.status_handling INFO: Allocated block ID 3763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3763 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3763 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3763.1709793962.7312074.submit parsl.executors.status_handling INFO: Allocated block ID 3764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3764 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3764 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3764.1709793962.7316368.submit parsl.executors.status_handling INFO: Allocated block ID 3765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3765 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3765 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3765.1709793962.7320683.submit parsl.executors.status_handling INFO: Allocated block ID 3766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3766 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3766 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3766.1709793962.7324903.submit parsl.executors.status_handling INFO: Allocated block ID 3767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3767 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3767 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3767.1709793962.7329154.submit parsl.executors.status_handling INFO: Allocated block ID 3768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3768 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3768 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3768.1709793962.7333493.submit parsl.executors.status_handling INFO: Allocated block ID 3769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3769 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3769 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3769.1709793962.7337747.submit parsl.executors.status_handling INFO: Allocated block ID 3770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3770 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3770 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3770.1709793962.7342005.submit parsl.executors.status_handling INFO: Allocated block ID 3771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3771 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3771 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3771.1709793962.7346227.submit parsl.executors.status_handling INFO: Allocated block ID 3772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3772 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3772 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3772.1709793962.7350469.submit parsl.executors.status_handling INFO: Allocated block ID 3773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3773 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3773 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3773.1709793962.7354722.submit parsl.executors.status_handling INFO: Allocated block ID 3774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3774 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3774 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3774.1709793962.7359047.submit parsl.executors.status_handling INFO: Allocated block ID 3775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3775 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3775 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3775.1709793962.7363346.submit parsl.executors.status_handling INFO: Allocated block ID 3776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3776 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3776 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3776.1709793962.7367513.submit parsl.executors.status_handling INFO: Allocated block ID 3777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3777 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3777 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3777.1709793962.7371843.submit parsl.executors.status_handling INFO: Allocated block ID 3778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3778 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3778 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3778.1709793962.7376082.submit parsl.executors.status_handling INFO: Allocated block ID 3779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3779 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3779 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3779.1709793962.7380302.submit parsl.executors.status_handling INFO: Allocated block ID 3780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3780 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3780 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3780.1709793962.7384424.submit parsl.executors.status_handling INFO: Allocated block ID 3781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3781 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3781 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3781.1709793962.738873.submit parsl.executors.status_handling INFO: Allocated block ID 3782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3782 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3782 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3782.1709793962.7392914.submit parsl.executors.status_handling INFO: Allocated block ID 3783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3783 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3783 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3783.1709793962.7397072.submit parsl.executors.status_handling INFO: Allocated block ID 3784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3784 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3784 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3784.1709793962.7401333.submit parsl.executors.status_handling INFO: Allocated block ID 3785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3785 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3785 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3785.1709793962.7405524.submit parsl.executors.status_handling INFO: Allocated block ID 3786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3786 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3786 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3786.1709793962.740988.submit parsl.executors.status_handling INFO: Allocated block ID 3787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3787 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3787 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3787.1709793962.7414143.submit parsl.executors.status_handling INFO: Allocated block ID 3788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3788 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3788 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3788.1709793962.741838.submit parsl.executors.status_handling INFO: Allocated block ID 3789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3789 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3789 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3789.1709793962.7422657.submit parsl.executors.status_handling INFO: Allocated block ID 3790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3790 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3790 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3790.1709793962.742681.submit parsl.executors.status_handling INFO: Allocated block ID 3791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3791 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3791 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3791.1709793962.7501676.submit parsl.executors.status_handling INFO: Allocated block ID 3792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3792 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3792 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3792.1709793962.7505991.submit parsl.executors.status_handling INFO: Allocated block ID 3793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3793 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3793 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3793.1709793962.75103.submit parsl.executors.status_handling INFO: Allocated block ID 3794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3794 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3794 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3794.1709793962.7514484.submit parsl.executors.status_handling INFO: Allocated block ID 3795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3795 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3795 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3795.1709793962.7518766.submit parsl.executors.status_handling INFO: Allocated block ID 3796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3796 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3796 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3796.1709793962.7522988.submit parsl.executors.status_handling INFO: Allocated block ID 3797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3797 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3797 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3797.1709793962.7527142.submit parsl.executors.status_handling INFO: Allocated block ID 3798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3798 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3798 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3798.1709793962.7531464.submit parsl.executors.status_handling INFO: Allocated block ID 3799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3799 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3799 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3799.1709793962.7535717.submit parsl.executors.status_handling INFO: Allocated block ID 3800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3800 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3800 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3800.1709793962.7540002.submit parsl.executors.status_handling INFO: Allocated block ID 3801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3801 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3801 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3801.1709793962.7544198.submit parsl.executors.status_handling INFO: Allocated block ID 3802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3802 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3802 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3802.1709793962.7548406.submit parsl.executors.status_handling INFO: Allocated block ID 3803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3803 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3803 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3803.1709793962.7552586.submit parsl.executors.status_handling INFO: Allocated block ID 3804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3804 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3804 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3804.1709793962.7556734.submit parsl.executors.status_handling INFO: Allocated block ID 3805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3805 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3805 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3805.1709793962.7561.submit parsl.executors.status_handling INFO: Allocated block ID 3806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3806 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3806 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3806.1709793962.756519.submit parsl.executors.status_handling INFO: Allocated block ID 3807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3807 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3807 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3807.1709793962.7569506.submit parsl.executors.status_handling INFO: Allocated block ID 3808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3808 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3808 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3808.1709793962.7573907.submit parsl.executors.status_handling INFO: Allocated block ID 3809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3809 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3809 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3809.1709793962.7578278.submit parsl.executors.status_handling INFO: Allocated block ID 3810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3810 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3810 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3810.1709793962.7583327.submit parsl.executors.status_handling INFO: Allocated block ID 3811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3811 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3811 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3811.1709793962.7588134.submit parsl.executors.status_handling INFO: Allocated block ID 3812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3812 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3812 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3812.1709793962.7592936.submit parsl.executors.status_handling INFO: Allocated block ID 3813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3813 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3813 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3813.1709793962.7597504.submit parsl.executors.status_handling INFO: Allocated block ID 3814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3814 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3814 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3814.1709793962.7601907.submit parsl.executors.status_handling INFO: Allocated block ID 3815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3815 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3815 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3815.1709793962.7606213.submit parsl.executors.status_handling INFO: Allocated block ID 3816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3816 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3816 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3816.1709793962.7610617.submit parsl.executors.status_handling INFO: Allocated block ID 3817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3817 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3817 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3817.1709793962.7614832.submit parsl.executors.status_handling INFO: Allocated block ID 3818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3818 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3818 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3818.1709793962.7619154.submit parsl.executors.status_handling INFO: Allocated block ID 3819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3819 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3819 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3819.1709793962.7623506.submit parsl.executors.status_handling INFO: Allocated block ID 3820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3820 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3820 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3820.1709793962.7627804.submit parsl.executors.status_handling INFO: Allocated block ID 3821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3821 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3821 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3821.1709793962.7763743.submit parsl.executors.status_handling INFO: Allocated block ID 3822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3822 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3822 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3822.1709793962.776819.submit parsl.executors.status_handling INFO: Allocated block ID 3823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3823 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3823 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3823.1709793962.7772586.submit parsl.executors.status_handling INFO: Allocated block ID 3824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3824 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3824 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3824.1709793962.7776835.submit parsl.executors.status_handling INFO: Allocated block ID 3825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3825 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3825 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3825.1709793962.7781143.submit parsl.executors.status_handling INFO: Allocated block ID 3826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3826 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3826 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3826.1709793962.7785335.submit parsl.executors.status_handling INFO: Allocated block ID 3827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3827 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3827 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3827.1709793962.7789538.submit parsl.executors.status_handling INFO: Allocated block ID 3828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3828 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3828 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3828.1709793962.7793746.submit parsl.executors.status_handling INFO: Allocated block ID 3829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3829 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3829 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3829.1709793962.7797968.submit parsl.executors.status_handling INFO: Allocated block ID 3830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3830 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3830 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3830.1709793962.7802231.submit parsl.executors.status_handling INFO: Allocated block ID 3831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3831 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3831 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3831.1709793962.780643.submit parsl.executors.status_handling INFO: Allocated block ID 3832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3832 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3832 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3832.1709793962.7810793.submit parsl.executors.status_handling INFO: Allocated block ID 3833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3833 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3833 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3833.1709793962.7815058.submit parsl.executors.status_handling INFO: Allocated block ID 3834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3834 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3834 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3834.1709793962.7819421.submit parsl.executors.status_handling INFO: Allocated block ID 3835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3835 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3835 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3835.1709793962.7823696.submit parsl.executors.status_handling INFO: Allocated block ID 3836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3836 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3836 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3836.1709793962.7827897.submit parsl.executors.status_handling INFO: Allocated block ID 3837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3837 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3837 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3837.1709793962.7832193.submit parsl.executors.status_handling INFO: Allocated block ID 3838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3838 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3838 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3838.1709793962.7836368.submit parsl.executors.status_handling INFO: Allocated block ID 3839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3839 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3839 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3839.1709793962.7840655.submit parsl.executors.status_handling INFO: Allocated block ID 3840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3840 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3840 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3840.1709793962.7844818.submit parsl.executors.status_handling INFO: Allocated block ID 3841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3841 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3841 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3841.1709793962.7849116.submit parsl.executors.status_handling INFO: Allocated block ID 3842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3842 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3842 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3842.1709793962.7853312.submit parsl.executors.status_handling INFO: Allocated block ID 3843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3843 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3843 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3843.1709793962.7857525.submit parsl.executors.status_handling INFO: Allocated block ID 3844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3844 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3844 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3844.1709793962.7861764.submit parsl.executors.status_handling INFO: Allocated block ID 3845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3845 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3845 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3845.1709793962.7865937.submit parsl.executors.status_handling INFO: Allocated block ID 3846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3846 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3846 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3846.1709793962.787046.submit parsl.executors.status_handling INFO: Allocated block ID 3847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3847 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3847 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3847.1709793962.78748.submit parsl.executors.status_handling INFO: Allocated block ID 3848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3848 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3848 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3848.1709793962.787914.submit parsl.executors.status_handling INFO: Allocated block ID 3849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3849 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3849 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3849.1709793962.7883477.submit parsl.executors.status_handling INFO: Allocated block ID 3850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3850 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3850 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3850.1709793962.788765.submit parsl.executors.status_handling INFO: Allocated block ID 3851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3851 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3851 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3851.1709793962.802333.submit parsl.executors.status_handling INFO: Allocated block ID 3852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3852 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3852 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3852.1709793962.8027737.submit parsl.executors.status_handling INFO: Allocated block ID 3853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3853 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3853 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3853.1709793962.8032343.submit parsl.executors.status_handling INFO: Allocated block ID 3854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3854 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3854 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3854.1709793962.8036735.submit parsl.executors.status_handling INFO: Allocated block ID 3855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3855 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3855 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3855.1709793962.804115.submit parsl.executors.status_handling INFO: Allocated block ID 3856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3856 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3856 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3856.1709793962.8045485.submit parsl.executors.status_handling INFO: Allocated block ID 3857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3857 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3857 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3857.1709793962.8049865.submit parsl.executors.status_handling INFO: Allocated block ID 3858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3858 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3858 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3858.1709793962.8054152.submit parsl.executors.status_handling INFO: Allocated block ID 3859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3859 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3859 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3859.1709793962.8058417.submit parsl.executors.status_handling INFO: Allocated block ID 3860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3860 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3860 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3860.1709793962.8062825.submit parsl.executors.status_handling INFO: Allocated block ID 3861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3861 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3861 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3861.1709793962.8067143.submit parsl.executors.status_handling INFO: Allocated block ID 3862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3862 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3862 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3862.1709793962.8071544.submit parsl.executors.status_handling INFO: Allocated block ID 3863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3863 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3863 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3863.1709793962.8075905.submit parsl.executors.status_handling INFO: Allocated block ID 3864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3864 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3864 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3864.1709793962.8080347.submit parsl.executors.status_handling INFO: Allocated block ID 3865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3865 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3865 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3865.1709793962.8084683.submit parsl.executors.status_handling INFO: Allocated block ID 3866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3866 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3866 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3866.1709793962.808899.submit parsl.executors.status_handling INFO: Allocated block ID 3867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3867 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3867 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3867.1709793962.809341.submit parsl.executors.status_handling INFO: Allocated block ID 3868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3868 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3868 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3868.1709793962.8097618.submit parsl.executors.status_handling INFO: Allocated block ID 3869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3869 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3869 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3869.1709793962.8101845.submit parsl.executors.status_handling INFO: Allocated block ID 3870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3870 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3870 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3870.1709793962.8106089.submit parsl.executors.status_handling INFO: Allocated block ID 3871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3871 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3871 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3871.1709793962.811059.submit parsl.executors.status_handling INFO: Allocated block ID 3872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3872 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3872 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3872.1709793962.8114903.submit parsl.executors.status_handling INFO: Allocated block ID 3873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3873 2024-03-06 22:46:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3873 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3873.1709793962.811922.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1346 parsl.dataflow.strategy DEBUG: Executor multi has 1346 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 3874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3874 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3874 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3874.1709793967.656013.submit parsl.executors.status_handling INFO: Allocated block ID 3875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3875 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3875 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3875.1709793967.6565073.submit parsl.executors.status_handling INFO: Allocated block ID 3876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3876 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3876 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3876.1709793967.6569507.submit parsl.executors.status_handling INFO: Allocated block ID 3877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3877 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3877 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3877.1709793967.6574.submit parsl.executors.status_handling INFO: Allocated block ID 3878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3878 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3878 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3878.1709793967.6578388.submit parsl.executors.status_handling INFO: Allocated block ID 3879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3879 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3879 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3879.1709793967.658278.submit parsl.executors.status_handling INFO: Allocated block ID 3880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3880 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3880 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3880.1709793967.6587207.submit parsl.executors.status_handling INFO: Allocated block ID 3881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3881 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3881 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3881.1709793967.6591592.submit parsl.executors.status_handling INFO: Allocated block ID 3882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3882 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3882 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3882.1709793967.6595845.submit parsl.executors.status_handling INFO: Allocated block ID 3883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3883 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3883 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3883.1709793967.6600187.submit parsl.executors.status_handling INFO: Allocated block ID 3884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3884 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3884 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3884.1709793967.6604447.submit parsl.executors.status_handling INFO: Allocated block ID 3885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3885 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3885 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3885.1709793967.6608717.submit parsl.executors.status_handling INFO: Allocated block ID 3886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3886 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3886 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3886.1709793967.6612968.submit parsl.executors.status_handling INFO: Allocated block ID 3887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3887 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3887 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3887.1709793967.661719.submit parsl.executors.status_handling INFO: Allocated block ID 3888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3888 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3888 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3888.1709793967.6621454.submit parsl.executors.status_handling INFO: Allocated block ID 3889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3889 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3889 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3889.1709793967.662567.submit parsl.executors.status_handling INFO: Allocated block ID 3890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3890 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3890 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3890.1709793967.6629906.submit parsl.executors.status_handling INFO: Allocated block ID 3891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3891 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3891 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3891.1709793967.6634133.submit parsl.executors.status_handling INFO: Allocated block ID 3892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3892 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3892 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3892.1709793967.6638472.submit parsl.executors.status_handling INFO: Allocated block ID 3893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3893 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3893 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3893.1709793967.664279.submit parsl.executors.status_handling INFO: Allocated block ID 3894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3894 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3894 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3894.1709793967.664703.submit parsl.executors.status_handling INFO: Allocated block ID 3895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3895 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3895 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3895.1709793967.6651368.submit parsl.executors.status_handling INFO: Allocated block ID 3896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3896 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3896 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3896.1709793967.6655633.submit parsl.executors.status_handling INFO: Allocated block ID 3897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3897 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3897 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3897.1709793967.6659982.submit parsl.executors.status_handling INFO: Allocated block ID 3898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3898 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3898 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3898.1709793967.66642.submit parsl.executors.status_handling INFO: Allocated block ID 3899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3899 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3899 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3899.1709793967.666859.submit parsl.executors.status_handling INFO: Allocated block ID 3900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3900 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3900 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3900.1709793967.667284.submit parsl.executors.status_handling INFO: Allocated block ID 3901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3901 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3901 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3901.1709793967.6677036.submit parsl.executors.status_handling INFO: Allocated block ID 3902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3902 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3902 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3902.1709793967.671253.submit parsl.executors.status_handling INFO: Allocated block ID 3903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3903 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3903 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3903.1709793967.6717076.submit parsl.executors.status_handling INFO: Allocated block ID 3904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3904 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3904 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3904.1709793967.6721485.submit parsl.executors.status_handling INFO: Allocated block ID 3905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3905 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3905 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3905.1709793967.6725776.submit parsl.executors.status_handling INFO: Allocated block ID 3906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3906 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3906 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3906.1709793967.6730113.submit parsl.executors.status_handling INFO: Allocated block ID 3907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3907 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3907 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3907.1709793967.6734335.submit parsl.executors.status_handling INFO: Allocated block ID 3908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3908 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3908 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3908.1709793967.6738586.submit parsl.executors.status_handling INFO: Allocated block ID 3909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3909 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3909 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3909.1709793967.674282.submit parsl.executors.status_handling INFO: Allocated block ID 3910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3910 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3910 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3910.1709793967.6747003.submit parsl.executors.status_handling INFO: Allocated block ID 3911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3911 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3911 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3911.1709793967.6751297.submit parsl.executors.status_handling INFO: Allocated block ID 3912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3912 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3912 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3912.1709793967.6755548.submit parsl.executors.status_handling INFO: Allocated block ID 3913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3913 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3913 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3913.1709793967.6759803.submit parsl.executors.status_handling INFO: Allocated block ID 3914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3914 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3914 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3914.1709793967.6764011.submit parsl.executors.status_handling INFO: Allocated block ID 3915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3915 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3915 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3915.1709793967.6768281.submit parsl.executors.status_handling INFO: Allocated block ID 3916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3916 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3916 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3916.1709793967.677258.submit parsl.executors.status_handling INFO: Allocated block ID 3917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3917 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3917 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3917.1709793967.677692.submit parsl.executors.status_handling INFO: Allocated block ID 3918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3918 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3918 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3918.1709793967.6781209.submit parsl.executors.status_handling INFO: Allocated block ID 3919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3919 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3919 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3919.1709793967.6785424.submit parsl.executors.status_handling INFO: Allocated block ID 3920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3920 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3920 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3920.1709793967.6789663.submit parsl.executors.status_handling INFO: Allocated block ID 3921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3921 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3921 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3921.1709793967.679386.submit parsl.executors.status_handling INFO: Allocated block ID 3922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3922 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3922 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3922.1709793967.6798053.submit parsl.executors.status_handling INFO: Allocated block ID 3923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3923 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3923 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3923.1709793967.6802363.submit parsl.executors.status_handling INFO: Allocated block ID 3924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3924 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3924 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3924.1709793967.6806538.submit parsl.executors.status_handling INFO: Allocated block ID 3925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3925 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3925 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3925.1709793967.6810827.submit parsl.executors.status_handling INFO: Allocated block ID 3926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3926 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3926 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3926.1709793967.6815012.submit parsl.executors.status_handling INFO: Allocated block ID 3927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3927 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3927 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3927.1709793967.6825395.submit parsl.executors.status_handling INFO: Allocated block ID 3928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3928 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3928 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3928.1709793967.6830373.submit parsl.executors.status_handling INFO: Allocated block ID 3929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3929 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3929 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3929.1709793967.6834853.submit parsl.executors.status_handling INFO: Allocated block ID 3930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3930 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3930 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3930.1709793967.6839242.submit parsl.executors.status_handling INFO: Allocated block ID 3931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3931 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3931 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3931.1709793967.6843605.submit parsl.executors.status_handling INFO: Allocated block ID 3932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3932 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3932 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3932.1709793967.6847794.submit parsl.executors.status_handling INFO: Allocated block ID 3933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3933 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3933 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3933.1709793967.685209.submit parsl.executors.status_handling INFO: Allocated block ID 3934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3934 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3934 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3934.1709793967.6856318.submit parsl.executors.status_handling INFO: Allocated block ID 3935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3935 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3935 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3935.1709793967.6860664.submit parsl.executors.status_handling INFO: Allocated block ID 3936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3936 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3936 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3936.1709793967.6864858.submit parsl.executors.status_handling INFO: Allocated block ID 3937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3937 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3937 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3937.1709793967.686917.submit parsl.executors.status_handling INFO: Allocated block ID 3938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3938 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3938 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3938.1709793967.6873472.submit parsl.executors.status_handling INFO: Allocated block ID 3939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3939 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3939 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3939.1709793967.687768.submit parsl.executors.status_handling INFO: Allocated block ID 3940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3940 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3940 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3940.1709793967.6882012.submit parsl.executors.status_handling INFO: Allocated block ID 3941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3941 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3941 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3941.1709793967.6886263.submit parsl.executors.status_handling INFO: Allocated block ID 3942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3942 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3942 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3942.1709793967.6890535.submit parsl.executors.status_handling INFO: Allocated block ID 3943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3943 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3943 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3943.1709793967.6894786.submit parsl.executors.status_handling INFO: Allocated block ID 3944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3944 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3944 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3944.1709793967.68991.submit parsl.executors.status_handling INFO: Allocated block ID 3945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3945 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3945 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3945.1709793967.6903346.submit parsl.executors.status_handling INFO: Allocated block ID 3946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3946 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3946 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3946.1709793967.6907537.submit parsl.executors.status_handling INFO: Allocated block ID 3947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3947 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3947 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3947.1709793967.6911817.submit parsl.executors.status_handling INFO: Allocated block ID 3948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3948 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3948 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3948.1709793967.6916003.submit parsl.executors.status_handling INFO: Allocated block ID 3949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3949 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3949 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3949.1709793967.6979384.submit parsl.executors.status_handling INFO: Allocated block ID 3950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3950 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3950 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3950.1709793967.6984127.submit parsl.executors.status_handling INFO: Allocated block ID 3951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3951 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3951 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3951.1709793967.6988626.submit parsl.executors.status_handling INFO: Allocated block ID 3952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3952 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3952 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3952.1709793967.6993127.submit parsl.executors.status_handling INFO: Allocated block ID 3953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3953 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3953 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3953.1709793967.6997397.submit parsl.executors.status_handling INFO: Allocated block ID 3954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3954 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3954 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3954.1709793967.7001712.submit parsl.executors.status_handling INFO: Allocated block ID 3955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3955 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3955 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3955.1709793967.7005935.submit parsl.executors.status_handling INFO: Allocated block ID 3956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3956 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3956 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3956.1709793967.7010202.submit parsl.executors.status_handling INFO: Allocated block ID 3957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3957 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3957 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3957.1709793967.7014449.submit parsl.executors.status_handling INFO: Allocated block ID 3958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3958 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3958 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3958.1709793967.7018738.submit parsl.executors.status_handling INFO: Allocated block ID 3959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3959 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3959 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3959.1709793967.7023654.submit parsl.executors.status_handling INFO: Allocated block ID 3960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3960 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3960 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3960.1709793967.7028193.submit parsl.executors.status_handling INFO: Allocated block ID 3961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3961 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3961 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3961.1709793967.7032611.submit parsl.executors.status_handling INFO: Allocated block ID 3962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3962 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3962 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3962.1709793967.7036934.submit parsl.executors.status_handling INFO: Allocated block ID 3963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3963 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3963 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3963.1709793967.704138.submit parsl.executors.status_handling INFO: Allocated block ID 3964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3964 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3964 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3964.1709793967.7045672.submit parsl.executors.status_handling INFO: Allocated block ID 3965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3965 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3965 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3965.1709793967.7049975.submit parsl.executors.status_handling INFO: Allocated block ID 3966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3966 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3966 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3966.1709793967.7054193.submit parsl.executors.status_handling INFO: Allocated block ID 3967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3967 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3967 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3967.1709793967.7058504.submit parsl.executors.status_handling INFO: Allocated block ID 3968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3968 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3968 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3968.1709793967.7062747.submit parsl.executors.status_handling INFO: Allocated block ID 3969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3969 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3969 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3969.1709793967.706692.submit parsl.executors.status_handling INFO: Allocated block ID 3970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3970 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3970 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3970.1709793967.707119.submit parsl.executors.status_handling INFO: Allocated block ID 3971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3971 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3971 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3971.1709793967.7075403.submit parsl.executors.status_handling INFO: Allocated block ID 3972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3972 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3972 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3972.1709793967.7082372.submit parsl.executors.status_handling INFO: Allocated block ID 3973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3973 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3973 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3973.1709793967.7086875.submit parsl.executors.status_handling INFO: Allocated block ID 3974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3974 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3974 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3974.1709793967.7091627.submit parsl.executors.status_handling INFO: Allocated block ID 3975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3975 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3975 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3975.1709793967.709623.submit parsl.executors.status_handling INFO: Allocated block ID 3976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3976 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3976 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3976.1709793967.710105.submit parsl.executors.status_handling INFO: Allocated block ID 3977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3977 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3977 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3977.1709793967.7105439.submit parsl.executors.status_handling INFO: Allocated block ID 3978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3978 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3978 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3978.1709793967.7109792.submit parsl.executors.status_handling INFO: Allocated block ID 3979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3979 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3979 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3979.1709793967.7114124.submit parsl.executors.status_handling INFO: Allocated block ID 3980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3980 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3980 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3980.1709793967.711868.submit parsl.executors.status_handling INFO: Allocated block ID 3981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3981 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3981 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3981.1709793967.712306.submit parsl.executors.status_handling INFO: Allocated block ID 3982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3982 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3982 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3982.1709793967.7127445.submit parsl.executors.status_handling INFO: Allocated block ID 3983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3983 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3983 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3983.1709793967.713174.submit parsl.executors.status_handling INFO: Allocated block ID 3984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3984 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3984 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3984.1709793967.7135932.submit parsl.executors.status_handling INFO: Allocated block ID 3985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3985 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3985 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3985.1709793967.7140422.submit parsl.executors.status_handling INFO: Allocated block ID 3986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3986 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3986 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3986.1709793967.7144718.submit parsl.executors.status_handling INFO: Allocated block ID 3987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3987 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3987 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3987.1709793967.7149005.submit parsl.executors.status_handling INFO: Allocated block ID 3988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3988 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3988 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3988.1709793967.7153265.submit parsl.executors.status_handling INFO: Allocated block ID 3989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3989 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3989 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3989.1709793967.715751.submit parsl.executors.status_handling INFO: Allocated block ID 3990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3990 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3990 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3990.1709793967.7161772.submit parsl.executors.status_handling INFO: Allocated block ID 3991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3991 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3991 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3991.1709793967.716598.submit parsl.executors.status_handling INFO: Allocated block ID 3992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3992 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3992 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3992.1709793967.717551.submit parsl.executors.status_handling INFO: Allocated block ID 3993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3993 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3993 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3993.1709793967.718033.submit parsl.executors.status_handling INFO: Allocated block ID 3994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3994 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3994 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3994.1709793967.7184741.submit parsl.executors.status_handling INFO: Allocated block ID 3995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3995 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3995 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3995.1709793967.7189271.submit parsl.executors.status_handling INFO: Allocated block ID 3996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3996 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3996 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3996.1709793967.7193496.submit parsl.executors.status_handling INFO: Allocated block ID 3997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3997 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3997 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3997.1709793967.7197704.submit parsl.executors.status_handling INFO: Allocated block ID 3998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3998 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3998 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3998.1709793967.7201982.submit parsl.executors.status_handling INFO: Allocated block ID 3999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3999 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3999 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=3999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-3999.1709793967.720622.submit parsl.executors.status_handling INFO: Allocated block ID 4000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4000 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4000 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4000.1709793967.721055.submit parsl.executors.status_handling INFO: Allocated block ID 4001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4001 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4001 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4001.1709793967.7214813.submit parsl.executors.status_handling INFO: Allocated block ID 4002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4002 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4002 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4002.1709793967.7219214.submit parsl.executors.status_handling INFO: Allocated block ID 4003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4003 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4003 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4003.1709793967.7223516.submit parsl.executors.status_handling INFO: Allocated block ID 4004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4004 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4004 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4004.1709793967.7227712.submit parsl.executors.status_handling INFO: Allocated block ID 4005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4005 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4005 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4005.1709793967.723227.submit parsl.executors.status_handling INFO: Allocated block ID 4006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4006 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4006 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4006.1709793967.7236598.submit parsl.executors.status_handling INFO: Allocated block ID 4007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4007 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4007 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4007.1709793967.7241132.submit parsl.executors.status_handling INFO: Allocated block ID 4008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4008 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4008 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4008.1709793967.7245312.submit parsl.executors.status_handling INFO: Allocated block ID 4009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4009 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4009 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4009.1709793967.7249541.submit parsl.executors.status_handling INFO: Allocated block ID 4010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4010 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4010 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4010.1709793967.7253745.submit parsl.executors.status_handling INFO: Allocated block ID 4011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4011 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4011 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4011.1709793967.7257946.submit parsl.executors.status_handling INFO: Allocated block ID 4012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4012 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4012 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4012.1709793967.7264292.submit parsl.executors.status_handling INFO: Allocated block ID 4013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4013 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4013 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4013.1709793967.7268698.submit parsl.executors.status_handling INFO: Allocated block ID 4014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4014 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4014 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4014.1709793967.727296.submit parsl.executors.status_handling INFO: Allocated block ID 4015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4015 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4015 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4015.1709793967.7277215.submit parsl.executors.status_handling INFO: Allocated block ID 4016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4016 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4016 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4016.1709793967.7281504.submit parsl.executors.status_handling INFO: Allocated block ID 4017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4017 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4017 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4017.1709793967.7285745.submit parsl.executors.status_handling INFO: Allocated block ID 4018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4018 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4018 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4018.1709793967.7290196.submit parsl.executors.status_handling INFO: Allocated block ID 4019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4019 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4019 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4019.1709793967.7294555.submit parsl.executors.status_handling INFO: Allocated block ID 4020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4020 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4020 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4020.1709793967.729893.submit parsl.executors.status_handling INFO: Allocated block ID 4021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4021 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4021 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4021.1709793967.730328.submit parsl.executors.status_handling INFO: Allocated block ID 4022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4022 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4022 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4022.1709793967.7307487.submit parsl.executors.status_handling INFO: Allocated block ID 4023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4023 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4023 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4023.1709793967.731181.submit parsl.executors.status_handling INFO: Allocated block ID 4024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4024 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4024 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4024.1709793967.7316055.submit parsl.executors.status_handling INFO: Allocated block ID 4025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4025 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4025 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4025.1709793967.7320356.submit parsl.executors.status_handling INFO: Allocated block ID 4026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4026 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4026 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4026.1709793967.7324576.submit parsl.executors.status_handling INFO: Allocated block ID 4027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4027 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4027 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4027.1709793967.7328815.submit parsl.executors.status_handling INFO: Allocated block ID 4028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4028 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4028 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4028.1709793967.7333057.submit parsl.executors.status_handling INFO: Allocated block ID 4029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4029 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4029 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4029.1709793967.733725.submit parsl.executors.status_handling INFO: Allocated block ID 4030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4030 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4030 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4030.1709793967.734155.submit parsl.executors.status_handling INFO: Allocated block ID 4031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4031 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4031 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4031.1709793967.7345796.submit parsl.executors.status_handling INFO: Allocated block ID 4032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4032 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4032 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4032.1709793967.7350736.submit parsl.executors.status_handling INFO: Allocated block ID 4033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4033 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4033 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4033.1709793967.7355065.submit parsl.executors.status_handling INFO: Allocated block ID 4034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4034 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4034 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4034.1709793967.735937.submit parsl.executors.status_handling INFO: Allocated block ID 4035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4035 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4035 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4035.1709793967.7363608.submit parsl.executors.status_handling INFO: Allocated block ID 4036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4036 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4036 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4036.1709793967.7367895.submit parsl.executors.status_handling INFO: Allocated block ID 4037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4037 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4037 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4037.1709793967.7372208.submit parsl.executors.status_handling INFO: Allocated block ID 4038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4038 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4038 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4038.1709793967.737643.submit parsl.executors.status_handling INFO: Allocated block ID 4039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4039 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4039 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4039.1709793967.7380702.submit parsl.executors.status_handling INFO: Allocated block ID 4040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4040 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4040 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4040.1709793967.7384908.submit parsl.executors.status_handling INFO: Allocated block ID 4041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4041 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4041 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4041.1709793967.7389193.submit parsl.executors.status_handling INFO: Allocated block ID 4042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4042 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4042 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4042.1709793967.73934.submit parsl.executors.status_handling INFO: Allocated block ID 4043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4043 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4043 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4043.1709793967.7397628.submit parsl.executors.status_handling INFO: Allocated block ID 4044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4044 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4044 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4044.1709793967.740196.submit parsl.executors.status_handling INFO: Allocated block ID 4045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4045 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4045 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4045.1709793967.7406144.submit parsl.executors.status_handling INFO: Allocated block ID 4046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4046 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4046 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4046.1709793967.7410696.submit parsl.executors.status_handling INFO: Allocated block ID 4047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4047 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4047 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4047.1709793967.741499.submit parsl.executors.status_handling INFO: Allocated block ID 4048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4048 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4048 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4048.1709793967.741937.submit parsl.executors.status_handling INFO: Allocated block ID 4049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4049 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4049 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4049.1709793967.7423673.submit parsl.executors.status_handling INFO: Allocated block ID 4050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4050 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4050 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4050.1709793967.7427886.submit parsl.executors.status_handling INFO: Allocated block ID 4051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4051 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4051 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4051.1709793967.7432172.submit parsl.executors.status_handling INFO: Allocated block ID 4052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4052 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4052 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4052.1709793967.7436607.submit parsl.executors.status_handling INFO: Allocated block ID 4053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4053 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4053 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4053.1709793967.7445512.submit parsl.executors.status_handling INFO: Allocated block ID 4054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4054 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4054 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4054.1709793967.7450228.submit parsl.executors.status_handling INFO: Allocated block ID 4055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4055 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4055 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4055.1709793967.7454674.submit parsl.executors.status_handling INFO: Allocated block ID 4056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4056 2024-03-06 22:46:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4056 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4056.1709793967.745913.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74f3b50c581b7b8387dc30124d8b9882 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a7649d44-ba96-403d-9980-058edf13514b_biasIsr_1123832_47.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/a7649d44-ba96-403d-9980-058edf13514b_biasIsr_1123832_47.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b348df112bfd52e75e89d5b066071224 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d843bbe8-07f0-414f-a15a-ad6a236ced25_biasIsr_1123839_47.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/d843bbe8-07f0-414f-a15a-ad6a236ced25_biasIsr_1123839_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1344 parsl.dataflow.strategy DEBUG: Executor multi has 1344 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4057 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4057 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4057.1709793972.6621678.submit parsl.executors.status_handling INFO: Allocated block ID 4058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4058 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4058 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4058.1709793972.6636636.submit parsl.executors.status_handling INFO: Allocated block ID 4059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4059 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4059 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4059.1709793972.6651208.submit parsl.executors.status_handling INFO: Allocated block ID 4060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4060 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4060 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4060.1709793972.6665425.submit parsl.executors.status_handling INFO: Allocated block ID 4061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4061 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4061 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4061.1709793972.6679626.submit parsl.executors.status_handling INFO: Allocated block ID 4062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4062 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4062 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4062.1709793972.669377.submit parsl.executors.status_handling INFO: Allocated block ID 4063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4063 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4063 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4063.1709793972.6707838.submit parsl.executors.status_handling INFO: Allocated block ID 4064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4064 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4064 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4064.1709793972.6713896.submit parsl.executors.status_handling INFO: Allocated block ID 4065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4065 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4065 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4065.1709793972.6720052.submit parsl.executors.status_handling INFO: Allocated block ID 4066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4066 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4066 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4066.1709793972.6726038.submit parsl.executors.status_handling INFO: Allocated block ID 4067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4067 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4067 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4067.1709793972.6732073.submit parsl.executors.status_handling INFO: Allocated block ID 4068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4068 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4068 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4068.1709793972.6738203.submit parsl.executors.status_handling INFO: Allocated block ID 4069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4069 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4069 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4069.1709793972.6744292.submit parsl.executors.status_handling INFO: Allocated block ID 4070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4070 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4070 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4070.1709793972.675037.submit parsl.executors.status_handling INFO: Allocated block ID 4071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4071 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4071 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4071.1709793972.675633.submit parsl.executors.status_handling INFO: Allocated block ID 4072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4072 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4072 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4072.1709793972.676234.submit parsl.executors.status_handling INFO: Allocated block ID 4073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4073 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4073 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4073.1709793972.6768308.submit parsl.executors.status_handling INFO: Allocated block ID 4074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4074 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4074 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4074.1709793972.6774182.submit parsl.executors.status_handling INFO: Allocated block ID 4075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4075 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4075 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4075.1709793972.6780272.submit parsl.executors.status_handling INFO: Allocated block ID 4076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4076 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4076 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4076.1709793972.678619.submit parsl.executors.status_handling INFO: Allocated block ID 4077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4077 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4077 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4077.1709793972.6792219.submit parsl.executors.status_handling INFO: Allocated block ID 4078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4078 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4078 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4078.1709793972.6798213.submit parsl.executors.status_handling INFO: Allocated block ID 4079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4079 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4079 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4079.1709793972.6804206.submit parsl.executors.status_handling INFO: Allocated block ID 4080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4080 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4080 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4080.1709793972.6810262.submit parsl.executors.status_handling INFO: Allocated block ID 4081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4081 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4081 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4081.1709793972.6814806.submit parsl.executors.status_handling INFO: Allocated block ID 4082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4082 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4082 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4082.1709793972.681884.submit parsl.executors.status_handling INFO: Allocated block ID 4083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4083 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4083 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4083.1709793972.6822815.submit parsl.executors.status_handling INFO: Allocated block ID 4084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4084 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4084 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4084.1709793972.682676.submit parsl.executors.status_handling INFO: Allocated block ID 4085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4085 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4085 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4085.1709793972.6831088.submit parsl.executors.status_handling INFO: Allocated block ID 4086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4086 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4086 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4086.1709793972.683528.submit parsl.executors.status_handling INFO: Allocated block ID 4087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4087 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4087 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4087.1709793972.6839685.submit parsl.executors.status_handling INFO: Allocated block ID 4088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4088 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4088 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4088.1709793972.6843903.submit parsl.executors.status_handling INFO: Allocated block ID 4089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4089 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4089 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4089.1709793972.6847944.submit parsl.executors.status_handling INFO: Allocated block ID 4090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4090 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4090 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4090.1709793972.6852334.submit parsl.executors.status_handling INFO: Allocated block ID 4091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4091 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4091 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4091.1709793972.6856592.submit parsl.executors.status_handling INFO: Allocated block ID 4092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4092 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4092 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4092.1709793972.6861403.submit parsl.executors.status_handling INFO: Allocated block ID 4093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4093 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4093 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4093.1709793972.6865704.submit parsl.executors.status_handling INFO: Allocated block ID 4094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4094 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4094 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4094.1709793972.686999.submit parsl.executors.status_handling INFO: Allocated block ID 4095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4095 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4095 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4095.1709793972.6874206.submit parsl.executors.status_handling INFO: Allocated block ID 4096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4096 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4096 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4096.1709793972.6878455.submit parsl.executors.status_handling INFO: Allocated block ID 4097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4097 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4097 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4097.1709793972.6882765.submit parsl.executors.status_handling INFO: Allocated block ID 4098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4098 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4098 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4098.1709793972.6886744.submit parsl.executors.status_handling INFO: Allocated block ID 4099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4099 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4099 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4099.1709793972.6890793.submit parsl.executors.status_handling INFO: Allocated block ID 4100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4100 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4100 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4100.1709793972.6894777.submit parsl.executors.status_handling INFO: Allocated block ID 4101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4101 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4101 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4101.1709793972.6898825.submit parsl.executors.status_handling INFO: Allocated block ID 4102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4102 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4102 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4102.1709793972.6902828.submit parsl.executors.status_handling INFO: Allocated block ID 4103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4103 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4103 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4103.1709793972.6906788.submit parsl.executors.status_handling INFO: Allocated block ID 4104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4104 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4104 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4104.1709793972.6910877.submit parsl.executors.status_handling INFO: Allocated block ID 4105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4105 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4105 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4105.1709793972.6914926.submit parsl.executors.status_handling INFO: Allocated block ID 4106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4106 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4106 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4106.1709793972.6918983.submit parsl.executors.status_handling INFO: Allocated block ID 4107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4107 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4107 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4107.1709793972.6922984.submit parsl.executors.status_handling INFO: Allocated block ID 4108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4108 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4108 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4108.1709793972.692702.submit parsl.executors.status_handling INFO: Allocated block ID 4109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4109 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4109 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4109.1709793972.6931064.submit parsl.executors.status_handling INFO: Allocated block ID 4110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4110 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4110 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4110.1709793972.6935065.submit parsl.executors.status_handling INFO: Allocated block ID 4111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4111 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4111 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4111.1709793972.6939166.submit parsl.executors.status_handling INFO: Allocated block ID 4112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4112 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4112 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4112.1709793972.6943173.submit parsl.executors.status_handling INFO: Allocated block ID 4113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4113 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4113 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4113.1709793972.694713.submit parsl.executors.status_handling INFO: Allocated block ID 4114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4114 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4114 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4114.1709793972.6951509.submit parsl.executors.status_handling INFO: Allocated block ID 4115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4115 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4115 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4115.1709793972.6955721.submit parsl.executors.status_handling INFO: Allocated block ID 4116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4116 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4116 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4116.1709793972.6960053.submit parsl.executors.status_handling INFO: Allocated block ID 4117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4117 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4117 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4117.1709793972.696433.submit parsl.executors.status_handling INFO: Allocated block ID 4118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4118 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4118 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4118.1709793972.6968653.submit parsl.executors.status_handling INFO: Allocated block ID 4119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4119 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4119 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4119.1709793972.6972837.submit parsl.executors.status_handling INFO: Allocated block ID 4120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4120 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4120 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4120.1709793972.6977026.submit parsl.executors.status_handling INFO: Allocated block ID 4121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4121 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4121 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4121.1709793972.6981297.submit parsl.executors.status_handling INFO: Allocated block ID 4122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4122 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4122 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4122.1709793972.6985495.submit parsl.executors.status_handling INFO: Allocated block ID 4123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4123 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4123 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4123.1709793972.6989763.submit parsl.executors.status_handling INFO: Allocated block ID 4124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4124 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4124 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4124.1709793972.6993737.submit parsl.executors.status_handling INFO: Allocated block ID 4125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4125 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4125 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4125.1709793972.6997747.submit parsl.executors.status_handling INFO: Allocated block ID 4126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4126 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4126 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4126.1709793972.700195.submit parsl.executors.status_handling INFO: Allocated block ID 4127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4127 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4127 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4127.1709793972.7006288.submit parsl.executors.status_handling INFO: Allocated block ID 4128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4128 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4128 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4128.1709793972.7011025.submit parsl.executors.status_handling INFO: Allocated block ID 4129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4129 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4129 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4129.1709793972.7015486.submit parsl.executors.status_handling INFO: Allocated block ID 4130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4130 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4130 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4130.1709793972.7019873.submit parsl.executors.status_handling INFO: Allocated block ID 4131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4131 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4131 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4131.1709793972.702421.submit parsl.executors.status_handling INFO: Allocated block ID 4132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4132 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4132 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4132.1709793972.7028458.submit parsl.executors.status_handling INFO: Allocated block ID 4133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4133 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4133 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4133.1709793972.7032592.submit parsl.executors.status_handling INFO: Allocated block ID 4134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4134 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4134 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4134.1709793972.7036557.submit parsl.executors.status_handling INFO: Allocated block ID 4135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4135 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4135 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4135.1709793972.7040637.submit parsl.executors.status_handling INFO: Allocated block ID 4136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4136 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4136 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4136.1709793972.7044585.submit parsl.executors.status_handling INFO: Allocated block ID 4137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4137 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4137 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4137.1709793972.7048655.submit parsl.executors.status_handling INFO: Allocated block ID 4138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4138 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4138 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4138.1709793972.7052643.submit parsl.executors.status_handling INFO: Allocated block ID 4139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4139 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4139 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4139.1709793972.7056565.submit parsl.executors.status_handling INFO: Allocated block ID 4140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4140 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4140 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4140.1709793972.706099.submit parsl.executors.status_handling INFO: Allocated block ID 4141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4141 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4141 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4141.1709793972.7065222.submit parsl.executors.status_handling INFO: Allocated block ID 4142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4142 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4142 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4142.1709793972.7069552.submit parsl.executors.status_handling INFO: Allocated block ID 4143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4143 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4143 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4143.1709793972.7073717.submit parsl.executors.status_handling INFO: Allocated block ID 4144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4144 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4144 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4144.1709793972.7077932.submit parsl.executors.status_handling INFO: Allocated block ID 4145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4145 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4145 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4145.1709793972.7082236.submit parsl.executors.status_handling INFO: Allocated block ID 4146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4146 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4146 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4146.1709793972.7086413.submit parsl.executors.status_handling INFO: Allocated block ID 4147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4147 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4147 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4147.1709793972.7090707.submit parsl.executors.status_handling INFO: Allocated block ID 4148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4148 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4148 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4148.1709793972.7094796.submit parsl.executors.status_handling INFO: Allocated block ID 4149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4149 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4149 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4149.1709793972.7099025.submit parsl.executors.status_handling INFO: Allocated block ID 4150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4150 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4150 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4150.1709793972.710331.submit parsl.executors.status_handling INFO: Allocated block ID 4151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4151 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4151 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4151.1709793972.7107723.submit parsl.executors.status_handling INFO: Allocated block ID 4152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4152 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4152 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4152.1709793972.7112312.submit parsl.executors.status_handling INFO: Allocated block ID 4153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4153 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4153 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4153.1709793972.711683.submit parsl.executors.status_handling INFO: Allocated block ID 4154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4154 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4154 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4154.1709793972.7121265.submit parsl.executors.status_handling INFO: Allocated block ID 4155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4155 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4155 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4155.1709793972.7125447.submit parsl.executors.status_handling INFO: Allocated block ID 4156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4156 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4156 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4156.1709793972.7129984.submit parsl.executors.status_handling INFO: Allocated block ID 4157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4157 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4157 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4157.1709793972.7134337.submit parsl.executors.status_handling INFO: Allocated block ID 4158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4158 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4158 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4158.1709793972.7138681.submit parsl.executors.status_handling INFO: Allocated block ID 4159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4159 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4159 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4159.1709793972.7143047.submit parsl.executors.status_handling INFO: Allocated block ID 4160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4160 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4160 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4160.1709793972.71472.submit parsl.executors.status_handling INFO: Allocated block ID 4161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4161 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4161 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4161.1709793972.7151427.submit parsl.executors.status_handling INFO: Allocated block ID 4162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4162 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4162 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4162.1709793972.7155592.submit parsl.executors.status_handling INFO: Allocated block ID 4163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4163 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4163 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4163.1709793972.715985.submit parsl.executors.status_handling INFO: Allocated block ID 4164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4164 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4164 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4164.1709793972.7164094.submit parsl.executors.status_handling INFO: Allocated block ID 4165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4165 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4165 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4165.1709793972.7168386.submit parsl.executors.status_handling INFO: Allocated block ID 4166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4166 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4166 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4166.1709793972.7172637.submit parsl.executors.status_handling INFO: Allocated block ID 4167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4167 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4167 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4167.1709793972.7176905.submit parsl.executors.status_handling INFO: Allocated block ID 4168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4168 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4168 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4168.1709793972.7181184.submit parsl.executors.status_handling INFO: Allocated block ID 4169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4169 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4169 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4169.1709793972.7185352.submit parsl.executors.status_handling INFO: Allocated block ID 4170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4170 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4170 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4170.1709793972.7203403.submit parsl.executors.status_handling INFO: Allocated block ID 4171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4171 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4171 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4171.1709793972.7207825.submit parsl.executors.status_handling INFO: Allocated block ID 4172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4172 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4172 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4172.1709793972.7212248.submit parsl.executors.status_handling INFO: Allocated block ID 4173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4173 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4173 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4173.1709793972.7216542.submit parsl.executors.status_handling INFO: Allocated block ID 4174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4174 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4174 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4174.1709793972.7220829.submit parsl.executors.status_handling INFO: Allocated block ID 4175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4175 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4175 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4175.1709793972.722504.submit parsl.executors.status_handling INFO: Allocated block ID 4176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4176 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4176 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4176.1709793972.7229383.submit parsl.executors.status_handling INFO: Allocated block ID 4177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4177 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4177 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4177.1709793972.7233677.submit parsl.executors.status_handling INFO: Allocated block ID 4178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4178 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4178 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4178.1709793972.7238045.submit parsl.executors.status_handling INFO: Allocated block ID 4179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4179 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4179 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4179.1709793972.7242482.submit parsl.executors.status_handling INFO: Allocated block ID 4180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4180 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4180 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4180.1709793972.7246807.submit parsl.executors.status_handling INFO: Allocated block ID 4181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4181 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4181 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4181.1709793972.7251196.submit parsl.executors.status_handling INFO: Allocated block ID 4182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4182 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4182 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4182.1709793972.725554.submit parsl.executors.status_handling INFO: Allocated block ID 4183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4183 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4183 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4183.1709793972.726009.submit parsl.executors.status_handling INFO: Allocated block ID 4184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4184 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4184 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4184.1709793972.726468.submit parsl.executors.status_handling INFO: Allocated block ID 4185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4185 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4185 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4185.1709793972.7269127.submit parsl.executors.status_handling INFO: Allocated block ID 4186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4186 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4186 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4186.1709793972.7273479.submit parsl.executors.status_handling INFO: Allocated block ID 4187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4187 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4187 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4187.1709793972.7278233.submit parsl.executors.status_handling INFO: Allocated block ID 4188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4188 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4188 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4188.1709793972.7282674.submit parsl.executors.status_handling INFO: Allocated block ID 4189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4189 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4189 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4189.1709793972.7287061.submit parsl.executors.status_handling INFO: Allocated block ID 4190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4190 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4190 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4190.1709793972.7291477.submit parsl.executors.status_handling INFO: Allocated block ID 4191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4191 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4191 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4191.1709793972.729568.submit parsl.executors.status_handling INFO: Allocated block ID 4192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4192 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4192 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4192.1709793972.7299962.submit parsl.executors.status_handling INFO: Allocated block ID 4193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4193 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4193 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4193.1709793972.7304225.submit parsl.executors.status_handling INFO: Allocated block ID 4194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4194 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4194 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4194.1709793972.7308471.submit parsl.executors.status_handling INFO: Allocated block ID 4195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4195 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4195 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4195.1709793972.7312694.submit parsl.executors.status_handling INFO: Allocated block ID 4196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4196 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4196 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4196.1709793972.7316651.submit parsl.executors.status_handling INFO: Allocated block ID 4197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4197 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4197 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4197.1709793972.7320726.submit parsl.executors.status_handling INFO: Allocated block ID 4198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4198 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4198 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4198.1709793972.7324722.submit parsl.executors.status_handling INFO: Allocated block ID 4199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4199 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4199 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4199.1709793972.7328765.submit parsl.executors.status_handling INFO: Allocated block ID 4200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4200 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4200 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4200.1709793972.7333045.submit parsl.executors.status_handling INFO: Allocated block ID 4201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4201 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4201 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4201.1709793972.7337325.submit parsl.executors.status_handling INFO: Allocated block ID 4202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4202 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4202 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4202.1709793972.7341907.submit parsl.executors.status_handling INFO: Allocated block ID 4203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4203 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4203 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4203.1709793972.7346196.submit parsl.executors.status_handling INFO: Allocated block ID 4204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4204 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4204 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4204.1709793972.7350647.submit parsl.executors.status_handling INFO: Allocated block ID 4205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4205 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4205 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4205.1709793972.7354836.submit parsl.executors.status_handling INFO: Allocated block ID 4206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4206 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4206 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4206.1709793972.7359023.submit parsl.executors.status_handling INFO: Allocated block ID 4207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4207 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4207 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4207.1709793972.73634.submit parsl.executors.status_handling INFO: Allocated block ID 4208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4208 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4208 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4208.1709793972.7367597.submit parsl.executors.status_handling INFO: Allocated block ID 4209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4209 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4209 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4209.1709793972.7371874.submit parsl.executors.status_handling INFO: Allocated block ID 4210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4210 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4210 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4210.1709793972.7376149.submit parsl.executors.status_handling INFO: Allocated block ID 4211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4211 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4211 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4211.1709793972.7380428.submit parsl.executors.status_handling INFO: Allocated block ID 4212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4212 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4212 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4212.1709793972.738475.submit parsl.executors.status_handling INFO: Allocated block ID 4213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4213 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4213 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4213.1709793972.7389078.submit parsl.executors.status_handling INFO: Allocated block ID 4214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4214 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4214 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4214.1709793972.7393317.submit parsl.executors.status_handling INFO: Allocated block ID 4215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4215 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4215 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4215.1709793972.7397509.submit parsl.executors.status_handling INFO: Allocated block ID 4216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4216 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4216 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4216.1709793972.740176.submit parsl.executors.status_handling INFO: Allocated block ID 4217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4217 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4217 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4217.1709793972.7405875.submit parsl.executors.status_handling INFO: Allocated block ID 4218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4218 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4218 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4218.1709793972.7409883.submit parsl.executors.status_handling INFO: Allocated block ID 4219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4219 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4219 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4219.1709793972.7413912.submit parsl.executors.status_handling INFO: Allocated block ID 4220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4220 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4220 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4220.1709793972.7417877.submit parsl.executors.status_handling INFO: Allocated block ID 4221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4221 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4221 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4221.1709793972.7421968.submit parsl.executors.status_handling INFO: Allocated block ID 4222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4222 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4222 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4222.1709793972.7425942.submit parsl.executors.status_handling INFO: Allocated block ID 4223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4223 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4223 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4223.1709793972.7430212.submit parsl.executors.status_handling INFO: Allocated block ID 4224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4224 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4224 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4224.1709793972.7434468.submit parsl.executors.status_handling INFO: Allocated block ID 4225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4225 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4225 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4225.1709793972.7438757.submit parsl.executors.status_handling INFO: Allocated block ID 4226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4226 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4226 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4226.1709793972.744297.submit parsl.executors.status_handling INFO: Allocated block ID 4227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4227 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4227 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4227.1709793972.744722.submit parsl.executors.status_handling INFO: Allocated block ID 4228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4228 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4228 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4228.1709793972.7451506.submit parsl.executors.status_handling INFO: Allocated block ID 4229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4229 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4229 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4229.1709793972.7455466.submit parsl.executors.status_handling INFO: Allocated block ID 4230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4230 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4230 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4230.1709793972.7459502.submit parsl.executors.status_handling INFO: Allocated block ID 4231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4231 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4231 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4231.1709793972.7463531.submit parsl.executors.status_handling INFO: Allocated block ID 4232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4232 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4232 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4232.1709793972.746779.submit parsl.executors.status_handling INFO: Allocated block ID 4233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4233 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4233 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4233.1709793972.747205.submit parsl.executors.status_handling INFO: Allocated block ID 4234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4234 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4234 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4234.1709793972.7476203.submit parsl.executors.status_handling INFO: Allocated block ID 4235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4235 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4235 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4235.1709793972.7480729.submit parsl.executors.status_handling INFO: Allocated block ID 4236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4236 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4236 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4236.1709793972.748498.submit parsl.executors.status_handling INFO: Allocated block ID 4237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4237 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4237 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4237.1709793972.7489302.submit parsl.executors.status_handling INFO: Allocated block ID 4238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4238 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4238 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4238.1709793972.749352.submit parsl.executors.status_handling INFO: Allocated block ID 4239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4239 2024-03-06 22:46:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4239 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4239.1709793972.749771.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95678c05bdcded74d4525faaf430787f with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7f9f12cb-a130-4f07-8df8-4c2ac2f3512d_biasIsr_1123733_47.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/7f9f12cb-a130-4f07-8df8-4c2ac2f3512d_biasIsr_1123733_47.stderr 2024-03-06 22:46:17 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1343 parsl.dataflow.strategy DEBUG: Executor multi has 1343 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4240 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4240 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4240.1709793977.7525668.submit parsl.executors.status_handling INFO: Allocated block ID 4241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4241 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4241 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4241.1709793977.7531369.submit parsl.executors.status_handling INFO: Allocated block ID 4242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4242 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4242 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4242.1709793977.7535887.submit parsl.executors.status_handling INFO: Allocated block ID 4243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4243 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4243 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4243.1709793977.7542198.submit parsl.executors.status_handling INFO: Allocated block ID 4244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4244 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4244 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4244.1709793977.7546666.submit parsl.executors.status_handling INFO: Allocated block ID 4245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4245 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4245 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4245.1709793977.7551143.submit parsl.executors.status_handling INFO: Allocated block ID 4246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4246 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4246 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4246.1709793977.7555606.submit parsl.executors.status_handling INFO: Allocated block ID 4247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4247 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4247 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4247.1709793977.7559962.submit parsl.executors.status_handling INFO: Allocated block ID 4248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4248 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4248 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4248.1709793977.7564273.submit parsl.executors.status_handling INFO: Allocated block ID 4249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4249 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4249 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4249.1709793977.7568676.submit parsl.executors.status_handling INFO: Allocated block ID 4250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4250 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4250 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4250.1709793977.7572975.submit parsl.executors.status_handling INFO: Allocated block ID 4251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4251 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4251 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4251.1709793977.757722.submit parsl.executors.status_handling INFO: Allocated block ID 4252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4252 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4252 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4252.1709793977.7581503.submit parsl.executors.status_handling INFO: Allocated block ID 4253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4253 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4253 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4253.1709793977.758569.submit parsl.executors.status_handling INFO: Allocated block ID 4254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4254 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4254 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4254.1709793977.7589953.submit parsl.executors.status_handling INFO: Allocated block ID 4255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4255 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4255 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4255.1709793977.7594156.submit parsl.executors.status_handling INFO: Allocated block ID 4256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4256 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4256 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4256.1709793977.75984.submit parsl.executors.status_handling INFO: Allocated block ID 4257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4257 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4257 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4257.1709793977.7602663.submit parsl.executors.status_handling INFO: Allocated block ID 4258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4258 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4258 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4258.1709793977.7606988.submit parsl.executors.status_handling INFO: Allocated block ID 4259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4259 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4259 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4259.1709793977.761137.submit parsl.executors.status_handling INFO: Allocated block ID 4260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4260 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4260 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4260.1709793977.7615943.submit parsl.executors.status_handling INFO: Allocated block ID 4261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4261 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4261 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4261.1709793977.762039.submit parsl.executors.status_handling INFO: Allocated block ID 4262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4262 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4262 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4262.1709793977.762482.submit parsl.executors.status_handling INFO: Allocated block ID 4263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4263 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4263 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4263.1709793977.7629068.submit parsl.executors.status_handling INFO: Allocated block ID 4264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4264 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4264 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4264.1709793977.7633302.submit parsl.executors.status_handling INFO: Allocated block ID 4265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4265 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4265 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4265.1709793977.7637498.submit parsl.executors.status_handling INFO: Allocated block ID 4266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4266 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4266 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4266.1709793977.764176.submit parsl.executors.status_handling INFO: Allocated block ID 4267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4267 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4267 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4267.1709793977.7645977.submit parsl.executors.status_handling INFO: Allocated block ID 4268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4268 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4268 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4268.1709793977.7762332.submit parsl.executors.status_handling INFO: Allocated block ID 4269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4269 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4269 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4269.1709793977.7767065.submit parsl.executors.status_handling INFO: Allocated block ID 4270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4270 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4270 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4270.1709793977.7771564.submit parsl.executors.status_handling INFO: Allocated block ID 4271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4271 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4271 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4271.1709793977.7775795.submit parsl.executors.status_handling INFO: Allocated block ID 4272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4272 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4272 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4272.1709793977.7780242.submit parsl.executors.status_handling INFO: Allocated block ID 4273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4273 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4273 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4273.1709793977.7784488.submit parsl.executors.status_handling INFO: Allocated block ID 4274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4274 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4274 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4274.1709793977.778875.submit parsl.executors.status_handling INFO: Allocated block ID 4275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4275 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4275 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4275.1709793977.7792962.submit parsl.executors.status_handling INFO: Allocated block ID 4276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4276 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4276 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4276.1709793977.779722.submit parsl.executors.status_handling INFO: Allocated block ID 4277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4277 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4277 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4277.1709793977.7801518.submit parsl.executors.status_handling INFO: Allocated block ID 4278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4278 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4278 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4278.1709793977.780582.submit parsl.executors.status_handling INFO: Allocated block ID 4279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4279 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4279 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4279.1709793977.7810066.submit parsl.executors.status_handling INFO: Allocated block ID 4280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4280 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4280 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4280.1709793977.7814286.submit parsl.executors.status_handling INFO: Allocated block ID 4281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4281 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4281 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4281.1709793977.7818546.submit parsl.executors.status_handling INFO: Allocated block ID 4282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4282 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4282 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4282.1709793977.7822797.submit parsl.executors.status_handling INFO: Allocated block ID 4283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4283 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4283 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4283.1709793977.7826989.submit parsl.executors.status_handling INFO: Allocated block ID 4284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4284 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4284 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4284.1709793977.7831287.submit parsl.executors.status_handling INFO: Allocated block ID 4285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4285 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4285 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4285.1709793977.7835517.submit parsl.executors.status_handling INFO: Allocated block ID 4286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4286 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4286 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4286.1709793977.7839768.submit parsl.executors.status_handling INFO: Allocated block ID 4287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4287 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4287 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4287.1709793977.7844021.submit parsl.executors.status_handling INFO: Allocated block ID 4288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4288 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4288 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4288.1709793977.7848313.submit parsl.executors.status_handling INFO: Allocated block ID 4289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4289 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4289 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4289.1709793977.7852526.submit parsl.executors.status_handling INFO: Allocated block ID 4290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4290 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4290 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4290.1709793977.7856688.submit parsl.executors.status_handling INFO: Allocated block ID 4291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4291 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4291 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4291.1709793977.7862368.submit parsl.executors.status_handling INFO: Allocated block ID 4292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4292 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4292 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4292.1709793977.786682.submit parsl.executors.status_handling INFO: Allocated block ID 4293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4293 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4293 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4293.1709793977.7871115.submit parsl.executors.status_handling INFO: Allocated block ID 4294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4294 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4294 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4294.1709793977.787536.submit parsl.executors.status_handling INFO: Allocated block ID 4295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4295 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4295 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4295.1709793977.7879605.submit parsl.executors.status_handling INFO: Allocated block ID 4296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4296 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4296 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4296.1709793977.7883875.submit parsl.executors.status_handling INFO: Allocated block ID 4297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4297 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4297 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4297.1709793977.7888174.submit parsl.executors.status_handling INFO: Allocated block ID 4298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4298 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4298 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4298.1709793977.7892387.submit parsl.executors.status_handling INFO: Allocated block ID 4299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4299 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4299 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4299.1709793977.7896616.submit parsl.executors.status_handling INFO: Allocated block ID 4300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4300 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4300 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4300.1709793977.79009.submit parsl.executors.status_handling INFO: Allocated block ID 4301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4301 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4301 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4301.1709793977.7905104.submit parsl.executors.status_handling INFO: Allocated block ID 4302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4302 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4302 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4302.1709793977.7909372.submit parsl.executors.status_handling INFO: Allocated block ID 4303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4303 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4303 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4303.1709793977.791357.submit parsl.executors.status_handling INFO: Allocated block ID 4304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4304 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4304 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4304.1709793977.7917745.submit parsl.executors.status_handling INFO: Allocated block ID 4305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4305 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4305 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4305.1709793977.7921996.submit parsl.executors.status_handling INFO: Allocated block ID 4306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4306 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4306 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4306.1709793977.7926216.submit parsl.executors.status_handling INFO: Allocated block ID 4307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4307 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4307 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4307.1709793977.7930725.submit parsl.executors.status_handling INFO: Allocated block ID 4308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4308 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4308 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4308.1709793977.7935011.submit parsl.executors.status_handling INFO: Allocated block ID 4309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4309 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4309 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4309.1709793977.7939303.submit parsl.executors.status_handling INFO: Allocated block ID 4310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4310 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4310 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4310.1709793977.7943473.submit parsl.executors.status_handling INFO: Allocated block ID 4311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4311 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4311 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4311.1709793977.7947612.submit parsl.executors.status_handling INFO: Allocated block ID 4312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4312 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4312 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4312.1709793977.7951906.submit parsl.executors.status_handling INFO: Allocated block ID 4313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4313 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4313 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4313.1709793977.7956114.submit parsl.executors.status_handling INFO: Allocated block ID 4314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4314 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4314 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4314.1709793977.8050842.submit parsl.executors.status_handling INFO: Allocated block ID 4315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4315 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4315 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4315.1709793977.8055758.submit parsl.executors.status_handling INFO: Allocated block ID 4316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4316 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4316 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4316.1709793977.8060188.submit parsl.executors.status_handling INFO: Allocated block ID 4317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4317 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4317 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4317.1709793977.8065023.submit parsl.executors.status_handling INFO: Allocated block ID 4318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4318 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4318 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4318.1709793977.8069367.submit parsl.executors.status_handling INFO: Allocated block ID 4319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4319 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4319 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4319.1709793977.8073623.submit parsl.executors.status_handling INFO: Allocated block ID 4320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4320 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4320 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4320.1709793977.807799.submit parsl.executors.status_handling INFO: Allocated block ID 4321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4321 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4321 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4321.1709793977.8082275.submit parsl.executors.status_handling INFO: Allocated block ID 4322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4322 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4322 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4322.1709793977.8086476.submit parsl.executors.status_handling INFO: Allocated block ID 4323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4323 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4323 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4323.1709793977.8090777.submit parsl.executors.status_handling INFO: Allocated block ID 4324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4324 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4324 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4324.1709793977.809502.submit parsl.executors.status_handling INFO: Allocated block ID 4325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4325 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4325 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4325.1709793977.8099287.submit parsl.executors.status_handling INFO: Allocated block ID 4326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4326 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4326 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4326.1709793977.8103507.submit parsl.executors.status_handling INFO: Allocated block ID 4327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4327 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4327 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4327.1709793977.810769.submit parsl.executors.status_handling INFO: Allocated block ID 4328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4328 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4328 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4328.1709793977.811194.submit parsl.executors.status_handling INFO: Allocated block ID 4329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4329 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4329 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4329.1709793977.8116112.submit parsl.executors.status_handling INFO: Allocated block ID 4330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4330 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4330 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4330.1709793977.812042.submit parsl.executors.status_handling INFO: Allocated block ID 4331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4331 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4331 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4331.1709793977.8124623.submit parsl.executors.status_handling INFO: Allocated block ID 4332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4332 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4332 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4332.1709793977.8128986.submit parsl.executors.status_handling INFO: Allocated block ID 4333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4333 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4333 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4333.1709793977.8133247.submit parsl.executors.status_handling INFO: Allocated block ID 4334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4334 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4334 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4334.1709793977.8137429.submit parsl.executors.status_handling INFO: Allocated block ID 4335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4335 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4335 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4335.1709793977.8141704.submit parsl.executors.status_handling INFO: Allocated block ID 4336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4336 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4336 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4336.1709793977.8145945.submit parsl.executors.status_handling INFO: Allocated block ID 4337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4337 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4337 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4337.1709793977.8239954.submit parsl.executors.status_handling INFO: Allocated block ID 4338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4338 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4338 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4338.1709793977.8244905.submit parsl.executors.status_handling INFO: Allocated block ID 4339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4339 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4339 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4339.1709793977.8249462.submit parsl.executors.status_handling INFO: Allocated block ID 4340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4340 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4340 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4340.1709793977.825507.submit parsl.executors.status_handling INFO: Allocated block ID 4341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4341 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4341 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4341.1709793977.8259518.submit parsl.executors.status_handling INFO: Allocated block ID 4342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4342 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4342 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4342.1709793977.8263862.submit parsl.executors.status_handling INFO: Allocated block ID 4343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4343 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4343 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4343.1709793977.8268216.submit parsl.executors.status_handling INFO: Allocated block ID 4344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4344 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4344 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4344.1709793977.827254.submit parsl.executors.status_handling INFO: Allocated block ID 4345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4345 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4345 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4345.1709793977.8276772.submit parsl.executors.status_handling INFO: Allocated block ID 4346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4346 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4346 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4346.1709793977.8281062.submit parsl.executors.status_handling INFO: Allocated block ID 4347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4347 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4347 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4347.1709793977.828532.submit parsl.executors.status_handling INFO: Allocated block ID 4348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4348 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4348 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4348.1709793977.828969.submit parsl.executors.status_handling INFO: Allocated block ID 4349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4349 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4349 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4349.1709793977.829394.submit parsl.executors.status_handling INFO: Allocated block ID 4350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4350 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4350 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4350.1709793977.829816.submit parsl.executors.status_handling INFO: Allocated block ID 4351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4351 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4351 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4351.1709793977.8302376.submit parsl.executors.status_handling INFO: Allocated block ID 4352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4352 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4352 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4352.1709793977.8306565.submit parsl.executors.status_handling INFO: Allocated block ID 4353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4353 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4353 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4353.1709793977.8310833.submit parsl.executors.status_handling INFO: Allocated block ID 4354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4354 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4354 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4354.1709793977.8315063.submit parsl.executors.status_handling INFO: Allocated block ID 4355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4355 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4355 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4355.1709793977.8319316.submit parsl.executors.status_handling INFO: Allocated block ID 4356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4356 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4356 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4356.1709793977.8323607.submit parsl.executors.status_handling INFO: Allocated block ID 4357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4357 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4357 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4357.1709793977.8327851.submit parsl.executors.status_handling INFO: Allocated block ID 4358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4358 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4358 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4358.1709793977.8332202.submit parsl.executors.status_handling INFO: Allocated block ID 4359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4359 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4359 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4359.1709793977.8336396.submit parsl.executors.status_handling INFO: Allocated block ID 4360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4360 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4360 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4360.1709793977.8431756.submit parsl.executors.status_handling INFO: Allocated block ID 4361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4361 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4361 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4361.1709793977.8436475.submit parsl.executors.status_handling INFO: Allocated block ID 4362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4362 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4362 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4362.1709793977.84414.submit parsl.executors.status_handling INFO: Allocated block ID 4363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4363 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4363 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4363.1709793977.8445804.submit parsl.executors.status_handling INFO: Allocated block ID 4364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4364 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4364 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4364.1709793977.8450148.submit parsl.executors.status_handling INFO: Allocated block ID 4365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4365 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4365 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4365.1709793977.845436.submit parsl.executors.status_handling INFO: Allocated block ID 4366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4366 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4366 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4366.1709793977.8458672.submit parsl.executors.status_handling INFO: Allocated block ID 4367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4367 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4367 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4367.1709793977.846293.submit parsl.executors.status_handling INFO: Allocated block ID 4368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4368 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4368 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4368.1709793977.8467224.submit parsl.executors.status_handling INFO: Allocated block ID 4369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4369 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4369 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4369.1709793977.8471546.submit parsl.executors.status_handling INFO: Allocated block ID 4370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4370 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4370 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4370.1709793977.8475745.submit parsl.executors.status_handling INFO: Allocated block ID 4371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4371 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4371 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4371.1709793977.8480005.submit parsl.executors.status_handling INFO: Allocated block ID 4372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4372 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4372 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4372.1709793977.8484273.submit parsl.executors.status_handling INFO: Allocated block ID 4373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4373 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4373 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4373.1709793977.8488538.submit parsl.executors.status_handling INFO: Allocated block ID 4374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4374 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4374 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4374.1709793977.8492749.submit parsl.executors.status_handling INFO: Allocated block ID 4375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4375 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4375 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4375.1709793977.8496933.submit parsl.executors.status_handling INFO: Allocated block ID 4376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4376 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4376 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4376.1709793977.8501194.submit parsl.executors.status_handling INFO: Allocated block ID 4377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4377 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4377 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4377.1709793977.8505476.submit parsl.executors.status_handling INFO: Allocated block ID 4378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4378 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4378 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4378.1709793977.8509812.submit parsl.executors.status_handling INFO: Allocated block ID 4379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4379 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4379 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4379.1709793977.8513987.submit parsl.executors.status_handling INFO: Allocated block ID 4380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4380 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4380 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4380.1709793977.8518293.submit parsl.executors.status_handling INFO: Allocated block ID 4381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4381 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4381 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4381.1709793977.8522556.submit parsl.executors.status_handling INFO: Allocated block ID 4382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4382 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4382 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4382.1709793977.8526785.submit parsl.executors.status_handling INFO: Allocated block ID 4383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4383 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4383 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4383.1709793977.8576102.submit parsl.executors.status_handling INFO: Allocated block ID 4384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4384 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4384 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4384.1709793977.858113.submit parsl.executors.status_handling INFO: Allocated block ID 4385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4385 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4385 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4385.1709793977.8585582.submit parsl.executors.status_handling INFO: Allocated block ID 4386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4386 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4386 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4386.1709793977.859007.submit parsl.executors.status_handling INFO: Allocated block ID 4387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4387 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4387 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4387.1709793977.8594322.submit parsl.executors.status_handling INFO: Allocated block ID 4388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4388 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4388 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4388.1709793977.8598638.submit parsl.executors.status_handling INFO: Allocated block ID 4389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4389 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4389 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4389.1709793977.8602843.submit parsl.executors.status_handling INFO: Allocated block ID 4390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4390 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4390 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4390.1709793977.8607087.submit parsl.executors.status_handling INFO: Allocated block ID 4391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4391 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4391 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4391.1709793977.861136.submit parsl.executors.status_handling INFO: Allocated block ID 4392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4392 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4392 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4392.1709793977.861568.submit parsl.executors.status_handling INFO: Allocated block ID 4393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4393 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4393 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4393.1709793977.8619995.submit parsl.executors.status_handling INFO: Allocated block ID 4394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4394 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4394 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4394.1709793977.8624237.submit parsl.executors.status_handling INFO: Allocated block ID 4395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4395 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4395 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4395.1709793977.862848.submit parsl.executors.status_handling INFO: Allocated block ID 4396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4396 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4396 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4396.1709793977.863275.submit parsl.executors.status_handling INFO: Allocated block ID 4397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4397 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4397 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4397.1709793977.8636973.submit parsl.executors.status_handling INFO: Allocated block ID 4398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4398 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4398 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4398.1709793977.8641253.submit parsl.executors.status_handling INFO: Allocated block ID 4399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4399 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4399 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4399.1709793977.8645434.submit parsl.executors.status_handling INFO: Allocated block ID 4400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4400 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4400 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4400.1709793977.864971.submit parsl.executors.status_handling INFO: Allocated block ID 4401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4401 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4401 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4401.1709793977.8653924.submit parsl.executors.status_handling INFO: Allocated block ID 4402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4402 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4402 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4402.1709793977.8658261.submit parsl.executors.status_handling INFO: Allocated block ID 4403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4403 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4403 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4403.1709793977.866246.submit parsl.executors.status_handling INFO: Allocated block ID 4404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4404 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4404 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4404.1709793977.866672.submit parsl.executors.status_handling INFO: Allocated block ID 4405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4405 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4405 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4405.1709793977.867106.submit parsl.executors.status_handling INFO: Allocated block ID 4406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4406 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4406 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4406.1709793977.8676887.submit parsl.executors.status_handling INFO: Allocated block ID 4407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4407 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4407 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4407.1709793977.8681417.submit parsl.executors.status_handling INFO: Allocated block ID 4408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4408 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4408 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4408.1709793977.868571.submit parsl.executors.status_handling INFO: Allocated block ID 4409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4409 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4409 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4409.1709793977.8690062.submit parsl.executors.status_handling INFO: Allocated block ID 4410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4410 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4410 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4410.1709793977.869425.submit parsl.executors.status_handling INFO: Allocated block ID 4411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4411 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4411 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4411.1709793977.8698497.submit parsl.executors.status_handling INFO: Allocated block ID 4412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4412 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4412 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4412.1709793977.8702686.submit parsl.executors.status_handling INFO: Allocated block ID 4413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4413 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4413 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4413.1709793977.870687.submit parsl.executors.status_handling INFO: Allocated block ID 4414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4414 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4414 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4414.1709793977.871114.submit parsl.executors.status_handling INFO: Allocated block ID 4415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4415 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4415 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4415.1709793977.8715622.submit parsl.executors.status_handling INFO: Allocated block ID 4416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4416 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4416 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4416.1709793977.8720174.submit parsl.executors.status_handling INFO: Allocated block ID 4417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4417 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4417 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4417.1709793977.872441.submit parsl.executors.status_handling INFO: Allocated block ID 4418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4418 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4418 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4418.1709793977.8728704.submit parsl.executors.status_handling INFO: Allocated block ID 4419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4419 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4419 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4419.1709793977.8732898.submit parsl.executors.status_handling INFO: Allocated block ID 4420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4420 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4420 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4420.1709793977.873715.submit parsl.executors.status_handling INFO: Allocated block ID 4421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4421 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4421 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4421.1709793977.8741367.submit parsl.executors.status_handling INFO: Allocated block ID 4422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4422 2024-03-06 22:46:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4422 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4422.1709793977.874556.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b4dea0dceafc60284066bda903b0ee with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 094014c01ff4be41740eb932398f1249 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1502cb578b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 1365 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/cpBiasCombine/6e3e0f02-3ba0-4a67-ae14-c384eed240cb_cpBiasCombine_47.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/cpBiasCombine/6e3e0f02-3ba0-4a67-ae14-c384eed240cb_cpBiasCombine_47.stderr parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/93658629-865d-4ad1-91fc-e4d48a54a3cc_biasIsr_1123834_47.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123834/93658629-865d-4ad1-91fc-e4d48a54a3cc_biasIsr_1123834_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1343 parsl.dataflow.strategy DEBUG: Executor multi has 1343 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4423 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4423 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4423.1709793982.6829333.submit parsl.executors.status_handling INFO: Allocated block ID 4424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4424 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4424 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4424.1709793982.6834455.submit parsl.executors.status_handling INFO: Allocated block ID 4425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4425 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4425 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4425.1709793982.6838932.submit parsl.executors.status_handling INFO: Allocated block ID 4426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4426 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4426 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4426.1709793982.684326.submit parsl.executors.status_handling INFO: Allocated block ID 4427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4427 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4427 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4427.1709793982.6847527.submit parsl.executors.status_handling INFO: Allocated block ID 4428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4428 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4428 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4428.1709793982.685192.submit parsl.executors.status_handling INFO: Allocated block ID 4429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4429 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4429 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4429.1709793982.6856186.submit parsl.executors.status_handling INFO: Allocated block ID 4430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4430 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4430 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4430.1709793982.6860797.submit parsl.executors.status_handling INFO: Allocated block ID 4431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4431 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4431 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4431.1709793982.6865032.submit parsl.executors.status_handling INFO: Allocated block ID 4432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4432 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4432 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4432.1709793982.6869378.submit parsl.executors.status_handling INFO: Allocated block ID 4433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4433 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4433 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4433.1709793982.6873608.submit parsl.executors.status_handling INFO: Allocated block ID 4434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4434 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4434 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4434.1709793982.6877854.submit parsl.executors.status_handling INFO: Allocated block ID 4435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4435 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4435 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4435.1709793982.6882236.submit parsl.executors.status_handling INFO: Allocated block ID 4436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4436 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4436 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4436.1709793982.6886492.submit parsl.executors.status_handling INFO: Allocated block ID 4437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4437 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4437 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4437.1709793982.689093.submit parsl.executors.status_handling INFO: Allocated block ID 4438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4438 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4438 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4438.1709793982.6895154.submit parsl.executors.status_handling INFO: Allocated block ID 4439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4439 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4439 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4439.1709793982.6899421.submit parsl.executors.status_handling INFO: Allocated block ID 4440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4440 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4440 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4440.1709793982.6903622.submit parsl.executors.status_handling INFO: Allocated block ID 4441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4441 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4441 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4441.1709793982.6907835.submit parsl.executors.status_handling INFO: Allocated block ID 4442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4442 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4442 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4442.1709793982.6912594.submit parsl.executors.status_handling INFO: Allocated block ID 4443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4443 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4443 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4443.1709793982.6916838.submit parsl.executors.status_handling INFO: Allocated block ID 4444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4444 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4444 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4444.1709793982.692114.submit parsl.executors.status_handling INFO: Allocated block ID 4445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4445 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4445 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4445.1709793982.6925354.submit parsl.executors.status_handling INFO: Allocated block ID 4446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4446 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4446 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4446.1709793982.6929646.submit parsl.executors.status_handling INFO: Allocated block ID 4447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4447 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4447 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4447.1709793982.6933947.submit parsl.executors.status_handling INFO: Allocated block ID 4448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4448 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4448 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4448.1709793982.6938272.submit parsl.executors.status_handling INFO: Allocated block ID 4449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4449 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4449 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4449.1709793982.6942523.submit parsl.executors.status_handling INFO: Allocated block ID 4450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4450 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4450 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4450.1709793982.6946769.submit parsl.executors.status_handling INFO: Allocated block ID 4451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4451 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4451 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4451.1709793982.6951065.submit parsl.executors.status_handling INFO: Allocated block ID 4452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4452 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4452 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4452.1709793982.6955326.submit parsl.executors.status_handling INFO: Allocated block ID 4453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4453 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4453 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4453.1709793982.6959708.submit parsl.executors.status_handling INFO: Allocated block ID 4454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4454 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4454 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4454.1709793982.696397.submit parsl.executors.status_handling INFO: Allocated block ID 4455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4455 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4455 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4455.1709793982.696824.submit parsl.executors.status_handling INFO: Allocated block ID 4456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4456 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4456 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4456.1709793982.6972477.submit parsl.executors.status_handling INFO: Allocated block ID 4457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4457 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4457 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4457.1709793982.6976721.submit parsl.executors.status_handling INFO: Allocated block ID 4458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4458 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4458 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4458.1709793982.6981032.submit parsl.executors.status_handling INFO: Allocated block ID 4459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4459 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4459 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4459.1709793982.69853.submit parsl.executors.status_handling INFO: Allocated block ID 4460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4460 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4460 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4460.1709793982.6989594.submit parsl.executors.status_handling INFO: Allocated block ID 4461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4461 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4461 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4461.1709793982.6993842.submit parsl.executors.status_handling INFO: Allocated block ID 4462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4462 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4462 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4462.1709793982.6998055.submit parsl.executors.status_handling INFO: Allocated block ID 4463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4463 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4463 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4463.1709793982.7003722.submit parsl.executors.status_handling INFO: Allocated block ID 4464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4464 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4464 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4464.1709793982.7007964.submit parsl.executors.status_handling INFO: Allocated block ID 4465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4465 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4465 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4465.1709793982.7012296.submit parsl.executors.status_handling INFO: Allocated block ID 4466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4466 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4466 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4466.1709793982.701649.submit parsl.executors.status_handling INFO: Allocated block ID 4467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4467 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4467 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4467.1709793982.7020786.submit parsl.executors.status_handling INFO: Allocated block ID 4468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4468 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4468 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4468.1709793982.7025.submit parsl.executors.status_handling INFO: Allocated block ID 4469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4469 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4469 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4469.1709793982.7029238.submit parsl.executors.status_handling INFO: Allocated block ID 4470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4470 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4470 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4470.1709793982.7033458.submit parsl.executors.status_handling INFO: Allocated block ID 4471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4471 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4471 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4471.1709793982.7037728.submit parsl.executors.status_handling INFO: Allocated block ID 4472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4472 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4472 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4472.1709793982.704211.submit parsl.executors.status_handling INFO: Allocated block ID 4473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4473 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4473 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4473.1709793982.7046351.submit parsl.executors.status_handling INFO: Allocated block ID 4474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4474 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4474 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4474.1709793982.705072.submit parsl.executors.status_handling INFO: Allocated block ID 4475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4475 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4475 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4475.1709793982.7054937.submit parsl.executors.status_handling INFO: Allocated block ID 4476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4476 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4476 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4476.1709793982.705917.submit parsl.executors.status_handling INFO: Allocated block ID 4477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4477 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4477 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4477.1709793982.7063355.submit parsl.executors.status_handling INFO: Allocated block ID 4478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4478 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4478 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4478.1709793982.7067554.submit parsl.executors.status_handling INFO: Allocated block ID 4479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4479 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4479 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4479.1709793982.7071867.submit parsl.executors.status_handling INFO: Allocated block ID 4480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4480 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4480 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4480.1709793982.7076075.submit parsl.executors.status_handling INFO: Allocated block ID 4481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4481 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4481 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4481.1709793982.7080367.submit parsl.executors.status_handling INFO: Allocated block ID 4482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4482 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4482 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4482.1709793982.7084584.submit parsl.executors.status_handling INFO: Allocated block ID 4483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4483 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4483 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4483.1709793982.7180903.submit parsl.executors.status_handling INFO: Allocated block ID 4484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4484 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4484 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4484.1709793982.7185693.submit parsl.executors.status_handling INFO: Allocated block ID 4485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4485 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4485 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4485.1709793982.719018.submit parsl.executors.status_handling INFO: Allocated block ID 4486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4486 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4486 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4486.1709793982.7194505.submit parsl.executors.status_handling INFO: Allocated block ID 4487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4487 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4487 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4487.1709793982.719882.submit parsl.executors.status_handling INFO: Allocated block ID 4488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4488 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4488 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4488.1709793982.7203085.submit parsl.executors.status_handling INFO: Allocated block ID 4489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4489 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4489 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4489.1709793982.720734.submit parsl.executors.status_handling INFO: Allocated block ID 4490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4490 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4490 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4490.1709793982.7211623.submit parsl.executors.status_handling INFO: Allocated block ID 4491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4491 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4491 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4491.1709793982.7215824.submit parsl.executors.status_handling INFO: Allocated block ID 4492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4492 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4492 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4492.1709793982.7220395.submit parsl.executors.status_handling INFO: Allocated block ID 4493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4493 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4493 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4493.1709793982.722473.submit parsl.executors.status_handling INFO: Allocated block ID 4494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4494 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4494 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4494.1709793982.7229242.submit parsl.executors.status_handling INFO: Allocated block ID 4495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4495 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4495 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4495.1709793982.7233763.submit parsl.executors.status_handling INFO: Allocated block ID 4496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4496 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4496 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4496.1709793982.7238176.submit parsl.executors.status_handling INFO: Allocated block ID 4497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4497 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4497 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4497.1709793982.724254.submit parsl.executors.status_handling INFO: Allocated block ID 4498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4498 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4498 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4498.1709793982.7246792.submit parsl.executors.status_handling INFO: Allocated block ID 4499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4499 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4499 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4499.1709793982.7251058.submit parsl.executors.status_handling INFO: Allocated block ID 4500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4500 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4500 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4500.1709793982.7255287.submit parsl.executors.status_handling INFO: Allocated block ID 4501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4501 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4501 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4501.1709793982.7259586.submit parsl.executors.status_handling INFO: Allocated block ID 4502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4502 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4502 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4502.1709793982.7263823.submit parsl.executors.status_handling INFO: Allocated block ID 4503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4503 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4503 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4503.1709793982.7268038.submit parsl.executors.status_handling INFO: Allocated block ID 4504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4504 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4504 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4504.1709793982.7364407.submit parsl.executors.status_handling INFO: Allocated block ID 4505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4505 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4505 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4505.1709793982.7368853.submit parsl.executors.status_handling INFO: Allocated block ID 4506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4506 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4506 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4506.1709793982.7373211.submit parsl.executors.status_handling INFO: Allocated block ID 4507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4507 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4507 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4507.1709793982.7377515.submit parsl.executors.status_handling INFO: Allocated block ID 4508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4508 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4508 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4508.1709793982.7381883.submit parsl.executors.status_handling INFO: Allocated block ID 4509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4509 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4509 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4509.1709793982.7386124.submit parsl.executors.status_handling INFO: Allocated block ID 4510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4510 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4510 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4510.1709793982.7390444.submit parsl.executors.status_handling INFO: Allocated block ID 4511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4511 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4511 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4511.1709793982.73947.submit parsl.executors.status_handling INFO: Allocated block ID 4512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4512 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4512 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4512.1709793982.7399044.submit parsl.executors.status_handling INFO: Allocated block ID 4513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4513 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4513 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4513.1709793982.7403302.submit parsl.executors.status_handling INFO: Allocated block ID 4514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4514 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4514 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4514.1709793982.7407548.submit parsl.executors.status_handling INFO: Allocated block ID 4515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4515 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4515 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4515.1709793982.7411826.submit parsl.executors.status_handling INFO: Allocated block ID 4516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4516 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4516 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4516.1709793982.741605.submit parsl.executors.status_handling INFO: Allocated block ID 4517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4517 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4517 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4517.1709793982.7420416.submit parsl.executors.status_handling INFO: Allocated block ID 4518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4518 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4518 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4518.1709793982.7424626.submit parsl.executors.status_handling INFO: Allocated block ID 4519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4519 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4519 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4519.1709793982.742895.submit parsl.executors.status_handling INFO: Allocated block ID 4520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4520 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4520 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4520.1709793982.7433188.submit parsl.executors.status_handling INFO: Allocated block ID 4521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4521 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4521 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4521.1709793982.743756.submit parsl.executors.status_handling INFO: Allocated block ID 4522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4522 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4522 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4522.1709793982.744193.submit parsl.executors.status_handling INFO: Allocated block ID 4523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4523 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4523 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4523.1709793982.7446249.submit parsl.executors.status_handling INFO: Allocated block ID 4524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4524 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4524 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4524.1709793982.7542312.submit parsl.executors.status_handling INFO: Allocated block ID 4525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4525 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4525 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4525.1709793982.754683.submit parsl.executors.status_handling INFO: Allocated block ID 4526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4526 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4526 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4526.1709793982.755126.submit parsl.executors.status_handling INFO: Allocated block ID 4527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4527 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4527 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4527.1709793982.7555583.submit parsl.executors.status_handling INFO: Allocated block ID 4528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4528 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4528 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4528.1709793982.7559924.submit parsl.executors.status_handling INFO: Allocated block ID 4529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4529 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4529 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4529.1709793982.756417.submit parsl.executors.status_handling INFO: Allocated block ID 4530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4530 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4530 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4530.1709793982.7568474.submit parsl.executors.status_handling INFO: Allocated block ID 4531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4531 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4531 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4531.1709793982.7572765.submit parsl.executors.status_handling INFO: Allocated block ID 4532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4532 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4532 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4532.1709793982.7577047.submit parsl.executors.status_handling INFO: Allocated block ID 4533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4533 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4533 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4533.1709793982.7581334.submit parsl.executors.status_handling INFO: Allocated block ID 4534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4534 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4534 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4534.1709793982.7585583.submit parsl.executors.status_handling INFO: Allocated block ID 4535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4535 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4535 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4535.1709793982.7589874.submit parsl.executors.status_handling INFO: Allocated block ID 4536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4536 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4536 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4536.1709793982.7594128.submit parsl.executors.status_handling INFO: Allocated block ID 4537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4537 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4537 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4537.1709793982.759841.submit parsl.executors.status_handling INFO: Allocated block ID 4538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4538 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4538 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4538.1709793982.7602699.submit parsl.executors.status_handling INFO: Allocated block ID 4539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4539 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4539 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4539.1709793982.760693.submit parsl.executors.status_handling INFO: Allocated block ID 4540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4540 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4540 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4540.1709793982.761122.submit parsl.executors.status_handling INFO: Allocated block ID 4541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4541 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4541 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4541.1709793982.761553.submit parsl.executors.status_handling INFO: Allocated block ID 4542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4542 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4542 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4542.1709793982.7619812.submit parsl.executors.status_handling INFO: Allocated block ID 4543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4543 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4543 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4543.1709793982.7624168.submit parsl.executors.status_handling INFO: Allocated block ID 4544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4544 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4544 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4544.1709793982.7628498.submit parsl.executors.status_handling INFO: Allocated block ID 4545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4545 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4545 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4545.1709793982.7632723.submit parsl.executors.status_handling INFO: Allocated block ID 4546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4546 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4546 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4546.1709793982.7636926.submit parsl.executors.status_handling INFO: Allocated block ID 4547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4547 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4547 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4547.1709793982.7642546.submit parsl.executors.status_handling INFO: Allocated block ID 4548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4548 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4548 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4548.1709793982.7646973.submit parsl.executors.status_handling INFO: Allocated block ID 4549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4549 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4549 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4549.1709793982.7651289.submit parsl.executors.status_handling INFO: Allocated block ID 4550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4550 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4550 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4550.1709793982.765555.submit parsl.executors.status_handling INFO: Allocated block ID 4551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4551 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4551 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4551.1709793982.7659862.submit parsl.executors.status_handling INFO: Allocated block ID 4552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4552 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4552 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4552.1709793982.7664104.submit parsl.executors.status_handling INFO: Allocated block ID 4553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4553 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4553 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4553.1709793982.7668402.submit parsl.executors.status_handling INFO: Allocated block ID 4554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4554 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4554 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4554.1709793982.7672682.submit parsl.executors.status_handling INFO: Allocated block ID 4555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4555 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4555 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4555.1709793982.7677002.submit parsl.executors.status_handling INFO: Allocated block ID 4556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4556 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4556 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4556.1709793982.7681296.submit parsl.executors.status_handling INFO: Allocated block ID 4557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4557 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4557 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4557.1709793982.7685587.submit parsl.executors.status_handling INFO: Allocated block ID 4558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4558 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4558 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4558.1709793982.7689924.submit parsl.executors.status_handling INFO: Allocated block ID 4559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4559 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4559 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4559.1709793982.7694201.submit parsl.executors.status_handling INFO: Allocated block ID 4560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4560 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4560 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4560.1709793982.7698455.submit parsl.executors.status_handling INFO: Allocated block ID 4561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4561 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4561 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4561.1709793982.7702675.submit parsl.executors.status_handling INFO: Allocated block ID 4562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4562 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4562 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4562.1709793982.770689.submit parsl.executors.status_handling INFO: Allocated block ID 4563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4563 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4563 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4563.1709793982.7711163.submit parsl.executors.status_handling INFO: Allocated block ID 4564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4564 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4564 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4564.1709793982.7715402.submit parsl.executors.status_handling INFO: Allocated block ID 4565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4565 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4565 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4565.1709793982.7719667.submit parsl.executors.status_handling INFO: Allocated block ID 4566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4566 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4566 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4566.1709793982.7723904.submit parsl.executors.status_handling INFO: Allocated block ID 4567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4567 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4567 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4567.1709793982.7728226.submit parsl.executors.status_handling INFO: Allocated block ID 4568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4568 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4568 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4568.1709793982.7732513.submit parsl.executors.status_handling INFO: Allocated block ID 4569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4569 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4569 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4569.1709793982.7736769.submit parsl.executors.status_handling INFO: Allocated block ID 4570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4570 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4570 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4570.1709793982.7741144.submit parsl.executors.status_handling INFO: Allocated block ID 4571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4571 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4571 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4571.1709793982.77455.submit parsl.executors.status_handling INFO: Allocated block ID 4572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4572 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4572 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4572.1709793982.775109.submit parsl.executors.status_handling INFO: Allocated block ID 4573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4573 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4573 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4573.1709793982.7755504.submit parsl.executors.status_handling INFO: Allocated block ID 4574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4574 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4574 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4574.1709793982.7759893.submit parsl.executors.status_handling INFO: Allocated block ID 4575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4575 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4575 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4575.1709793982.7764127.submit parsl.executors.status_handling INFO: Allocated block ID 4576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4576 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4576 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4576.1709793982.7768419.submit parsl.executors.status_handling INFO: Allocated block ID 4577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4577 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4577 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4577.1709793982.7772653.submit parsl.executors.status_handling INFO: Allocated block ID 4578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4578 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4578 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4578.1709793982.7776866.submit parsl.executors.status_handling INFO: Allocated block ID 4579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4579 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4579 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4579.1709793982.7781262.submit parsl.executors.status_handling INFO: Allocated block ID 4580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4580 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4580 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4580.1709793982.778552.submit parsl.executors.status_handling INFO: Allocated block ID 4581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4581 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4581 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4581.1709793982.7789886.submit parsl.executors.status_handling INFO: Allocated block ID 4582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4582 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4582 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4582.1709793982.7794154.submit parsl.executors.status_handling INFO: Allocated block ID 4583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4583 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4583 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4583.1709793982.7798438.submit parsl.executors.status_handling INFO: Allocated block ID 4584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4584 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4584 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4584.1709793982.7802668.submit parsl.executors.status_handling INFO: Allocated block ID 4585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4585 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4585 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4585.1709793982.780689.submit parsl.executors.status_handling INFO: Allocated block ID 4586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4586 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4586 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4586.1709793982.7811184.submit parsl.executors.status_handling INFO: Allocated block ID 4587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4587 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4587 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4587.1709793982.7815447.submit parsl.executors.status_handling INFO: Allocated block ID 4588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4588 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4588 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4588.1709793982.781976.submit parsl.executors.status_handling INFO: Allocated block ID 4589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4589 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4589 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4589.1709793982.7823994.submit parsl.executors.status_handling INFO: Allocated block ID 4590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4590 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4590 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4590.1709793982.7828267.submit parsl.executors.status_handling INFO: Allocated block ID 4591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4591 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4591 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4591.1709793982.7832515.submit parsl.executors.status_handling INFO: Allocated block ID 4592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4592 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4592 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4592.1709793982.783678.submit parsl.executors.status_handling INFO: Allocated block ID 4593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4593 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4593 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4593.1709793982.784107.submit parsl.executors.status_handling INFO: Allocated block ID 4594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4594 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4594 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4594.1709793982.7845273.submit parsl.executors.status_handling INFO: Allocated block ID 4595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4595 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4595 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4595.1709793982.784959.submit parsl.executors.status_handling INFO: Allocated block ID 4596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4596 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4596 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4596.1709793982.7853832.submit parsl.executors.status_handling INFO: Allocated block ID 4597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4597 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4597 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4597.1709793982.7858038.submit parsl.executors.status_handling INFO: Allocated block ID 4598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4598 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4598 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4598.1709793982.7863686.submit parsl.executors.status_handling INFO: Allocated block ID 4599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4599 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4599 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4599.1709793982.7867956.submit parsl.executors.status_handling INFO: Allocated block ID 4600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4600 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4600 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4600.1709793982.787225.submit parsl.executors.status_handling INFO: Allocated block ID 4601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4601 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4601 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4601.1709793982.787648.submit parsl.executors.status_handling INFO: Allocated block ID 4602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4602 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4602 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4602.1709793982.7880812.submit parsl.executors.status_handling INFO: Allocated block ID 4603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4603 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4603 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4603.1709793982.7885146.submit parsl.executors.status_handling INFO: Allocated block ID 4604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4604 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4604 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4604.1709793982.7889469.submit parsl.executors.status_handling INFO: Allocated block ID 4605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4605 2024-03-06 22:46:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4605 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4605.1709793982.789375.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1343 parsl.dataflow.strategy DEBUG: Executor multi has 1343 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4606 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4606 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4606.1709793987.6653864.submit parsl.executors.status_handling INFO: Allocated block ID 4607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4607 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4607 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4607.1709793987.6658852.submit parsl.executors.status_handling INFO: Allocated block ID 4608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4608 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4608 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4608.1709793987.6663241.submit parsl.executors.status_handling INFO: Allocated block ID 4609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4609 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4609 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4609.1709793987.6667523.submit parsl.executors.status_handling INFO: Allocated block ID 4610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4610 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4610 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4610.1709793987.667186.submit parsl.executors.status_handling INFO: Allocated block ID 4611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4611 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4611 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4611.1709793987.6676116.submit parsl.executors.status_handling INFO: Allocated block ID 4612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4612 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4612 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4612.1709793987.6680527.submit parsl.executors.status_handling INFO: Allocated block ID 4613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4613 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4613 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4613.1709793987.6684885.submit parsl.executors.status_handling INFO: Allocated block ID 4614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4614 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4614 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4614.1709793987.6689246.submit parsl.executors.status_handling INFO: Allocated block ID 4615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4615 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4615 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4615.1709793987.6693468.submit parsl.executors.status_handling INFO: Allocated block ID 4616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4616 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4616 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4616.1709793987.6697671.submit parsl.executors.status_handling INFO: Allocated block ID 4617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4617 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4617 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4617.1709793987.6702003.submit parsl.executors.status_handling INFO: Allocated block ID 4618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4618 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4618 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4618.1709793987.6706264.submit parsl.executors.status_handling INFO: Allocated block ID 4619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4619 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4619 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4619.1709793987.6710527.submit parsl.executors.status_handling INFO: Allocated block ID 4620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4620 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4620 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4620.1709793987.6714776.submit parsl.executors.status_handling INFO: Allocated block ID 4621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4621 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4621 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4621.1709793987.671906.submit parsl.executors.status_handling INFO: Allocated block ID 4622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4622 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4622 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4622.1709793987.672326.submit parsl.executors.status_handling INFO: Allocated block ID 4623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4623 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4623 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4623.1709793987.6727543.submit parsl.executors.status_handling INFO: Allocated block ID 4624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4624 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4624 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4624.1709793987.6731882.submit parsl.executors.status_handling INFO: Allocated block ID 4625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4625 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4625 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4625.1709793987.673619.submit parsl.executors.status_handling INFO: Allocated block ID 4626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4626 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4626 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4626.1709793987.6740475.submit parsl.executors.status_handling INFO: Allocated block ID 4627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4627 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4627 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4627.1709793987.6744707.submit parsl.executors.status_handling INFO: Allocated block ID 4628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4628 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4628 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4628.1709793987.6749015.submit parsl.executors.status_handling INFO: Allocated block ID 4629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4629 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4629 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4629.1709793987.6753247.submit parsl.executors.status_handling INFO: Allocated block ID 4630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4630 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4630 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4630.1709793987.6757505.submit parsl.executors.status_handling INFO: Allocated block ID 4631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4631 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4631 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4631.1709793987.67618.submit parsl.executors.status_handling INFO: Allocated block ID 4632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4632 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4632 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4632.1709793987.6766071.submit parsl.executors.status_handling INFO: Allocated block ID 4633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4633 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4633 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4633.1709793987.6770356.submit parsl.executors.status_handling INFO: Allocated block ID 4634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4634 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4634 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4634.1709793987.6774604.submit parsl.executors.status_handling INFO: Allocated block ID 4635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4635 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4635 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4635.1709793987.6910055.submit parsl.executors.status_handling INFO: Allocated block ID 4636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4636 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4636 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4636.1709793987.6915078.submit parsl.executors.status_handling INFO: Allocated block ID 4637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4637 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4637 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4637.1709793987.6919584.submit parsl.executors.status_handling INFO: Allocated block ID 4638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4638 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4638 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4638.1709793987.6923919.submit parsl.executors.status_handling INFO: Allocated block ID 4639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4639 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4639 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4639.1709793987.6928215.submit parsl.executors.status_handling INFO: Allocated block ID 4640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4640 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4640 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4640.1709793987.6932454.submit parsl.executors.status_handling INFO: Allocated block ID 4641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4641 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4641 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4641.1709793987.693666.submit parsl.executors.status_handling INFO: Allocated block ID 4642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4642 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4642 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4642.1709793987.6940985.submit parsl.executors.status_handling INFO: Allocated block ID 4643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4643 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4643 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4643.1709793987.694519.submit parsl.executors.status_handling INFO: Allocated block ID 4644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4644 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4644 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4644.1709793987.694953.submit parsl.executors.status_handling INFO: Allocated block ID 4645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4645 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4645 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4645.1709793987.6953735.submit parsl.executors.status_handling INFO: Allocated block ID 4646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4646 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4646 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4646.1709793987.6957967.submit parsl.executors.status_handling INFO: Allocated block ID 4647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4647 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4647 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4647.1709793987.6962283.submit parsl.executors.status_handling INFO: Allocated block ID 4648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4648 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4648 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4648.1709793987.696654.submit parsl.executors.status_handling INFO: Allocated block ID 4649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4649 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4649 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4649.1709793987.6970887.submit parsl.executors.status_handling INFO: Allocated block ID 4650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4650 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4650 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4650.1709793987.697509.submit parsl.executors.status_handling INFO: Allocated block ID 4651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4651 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4651 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4651.1709793987.6979358.submit parsl.executors.status_handling INFO: Allocated block ID 4652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4652 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4652 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4652.1709793987.6983573.submit parsl.executors.status_handling INFO: Allocated block ID 4653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4653 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4653 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4653.1709793987.6987762.submit parsl.executors.status_handling INFO: Allocated block ID 4654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4654 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4654 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4654.1709793987.6992083.submit parsl.executors.status_handling INFO: Allocated block ID 4655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4655 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4655 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4655.1709793987.6996286.submit parsl.executors.status_handling INFO: Allocated block ID 4656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4656 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4656 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4656.1709793987.700055.submit parsl.executors.status_handling INFO: Allocated block ID 4657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4657 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4657 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4657.1709793987.7004743.submit parsl.executors.status_handling INFO: Allocated block ID 4658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4658 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4658 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4658.1709793987.7009006.submit parsl.executors.status_handling INFO: Allocated block ID 4659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4659 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4659 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4659.1709793987.7013245.submit parsl.executors.status_handling INFO: Allocated block ID 4660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4660 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4660 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4660.1709793987.7017467.submit parsl.executors.status_handling INFO: Allocated block ID 4661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4661 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4661 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4661.1709793987.7021766.submit parsl.executors.status_handling INFO: Allocated block ID 4662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4662 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4662 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4662.1709793987.7026007.submit parsl.executors.status_handling INFO: Allocated block ID 4663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4663 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4663 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4663.1709793987.7030325.submit parsl.executors.status_handling INFO: Allocated block ID 4664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4664 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4664 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4664.1709793987.7034562.submit parsl.executors.status_handling INFO: Allocated block ID 4665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4665 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4665 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4665.1709793987.7170284.submit parsl.executors.status_handling INFO: Allocated block ID 4666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4666 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4666 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4666.1709793987.7175694.submit parsl.executors.status_handling INFO: Allocated block ID 4667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4667 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4667 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4667.1709793987.718015.submit parsl.executors.status_handling INFO: Allocated block ID 4668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4668 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4668 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4668.1709793987.718445.submit parsl.executors.status_handling INFO: Allocated block ID 4669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4669 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4669 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4669.1709793987.7188764.submit parsl.executors.status_handling INFO: Allocated block ID 4670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4670 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4670 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4670.1709793987.719302.submit parsl.executors.status_handling INFO: Allocated block ID 4671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4671 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4671 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4671.1709793987.7197206.submit parsl.executors.status_handling INFO: Allocated block ID 4672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4672 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4672 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4672.1709793987.7201545.submit parsl.executors.status_handling INFO: Allocated block ID 4673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4673 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4673 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4673.1709793987.7206082.submit parsl.executors.status_handling INFO: Allocated block ID 4674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4674 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4674 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4674.1709793987.721051.submit parsl.executors.status_handling INFO: Allocated block ID 4675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4675 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4675 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4675.1709793987.7214885.submit parsl.executors.status_handling INFO: Allocated block ID 4676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4676 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4676 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4676.1709793987.7219272.submit parsl.executors.status_handling INFO: Allocated block ID 4677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4677 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4677 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4677.1709793987.7223518.submit parsl.executors.status_handling INFO: Allocated block ID 4678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4678 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4678 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4678.1709793987.7227745.submit parsl.executors.status_handling INFO: Allocated block ID 4679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4679 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4679 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4679.1709793987.7232041.submit parsl.executors.status_handling INFO: Allocated block ID 4680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4680 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4680 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4680.1709793987.723622.submit parsl.executors.status_handling INFO: Allocated block ID 4681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4681 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4681 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4681.1709793987.7240484.submit parsl.executors.status_handling INFO: Allocated block ID 4682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4682 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4682 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4682.1709793987.7244706.submit parsl.executors.status_handling INFO: Allocated block ID 4683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4683 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4683 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4683.1709793987.7248938.submit parsl.executors.status_handling INFO: Allocated block ID 4684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4684 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4684 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4684.1709793987.7253177.submit parsl.executors.status_handling INFO: Allocated block ID 4685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4685 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4685 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4685.1709793987.725743.submit parsl.executors.status_handling INFO: Allocated block ID 4686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4686 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4686 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4686.1709793987.7261724.submit parsl.executors.status_handling INFO: Allocated block ID 4687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4687 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4687 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4687.1709793987.7265995.submit parsl.executors.status_handling INFO: Allocated block ID 4688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4688 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4688 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4688.1709793987.7270284.submit parsl.executors.status_handling INFO: Allocated block ID 4689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4689 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4689 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4689.1709793987.7274497.submit parsl.executors.status_handling INFO: Allocated block ID 4690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4690 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4690 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4690.1709793987.7278824.submit parsl.executors.status_handling INFO: Allocated block ID 4691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4691 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4691 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4691.1709793987.7283037.submit parsl.executors.status_handling INFO: Allocated block ID 4692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4692 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4692 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4692.1709793987.7287285.submit parsl.executors.status_handling INFO: Allocated block ID 4693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4693 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4693 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4693.1709793987.7291558.submit parsl.executors.status_handling INFO: Allocated block ID 4694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4694 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4694 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4694.1709793987.7295725.submit parsl.executors.status_handling INFO: Allocated block ID 4695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4695 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4695 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4695.1709793987.743132.submit parsl.executors.status_handling INFO: Allocated block ID 4696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4696 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4696 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4696.1709793987.743632.submit parsl.executors.status_handling INFO: Allocated block ID 4697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4697 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4697 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4697.1709793987.7440755.submit parsl.executors.status_handling INFO: Allocated block ID 4698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4698 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4698 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4698.1709793987.7445061.submit parsl.executors.status_handling INFO: Allocated block ID 4699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4699 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4699 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4699.1709793987.7449398.submit parsl.executors.status_handling INFO: Allocated block ID 4700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4700 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4700 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4700.1709793987.7453656.submit parsl.executors.status_handling INFO: Allocated block ID 4701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4701 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4701 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4701.1709793987.745785.submit parsl.executors.status_handling INFO: Allocated block ID 4702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4702 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4702 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4702.1709793987.7462187.submit parsl.executors.status_handling INFO: Allocated block ID 4703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4703 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4703 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4703.1709793987.7466388.submit parsl.executors.status_handling INFO: Allocated block ID 4704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4704 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4704 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4704.1709793987.7470655.submit parsl.executors.status_handling INFO: Allocated block ID 4705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4705 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4705 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4705.1709793987.7474878.submit parsl.executors.status_handling INFO: Allocated block ID 4706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4706 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4706 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4706.1709793987.7479193.submit parsl.executors.status_handling INFO: Allocated block ID 4707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4707 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4707 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4707.1709793987.7483463.submit parsl.executors.status_handling INFO: Allocated block ID 4708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4708 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4708 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4708.1709793987.7487705.submit parsl.executors.status_handling INFO: Allocated block ID 4709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4709 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4709 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4709.1709793987.7492082.submit parsl.executors.status_handling INFO: Allocated block ID 4710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4710 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4710 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4710.1709793987.7496293.submit parsl.executors.status_handling INFO: Allocated block ID 4711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4711 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4711 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4711.1709793987.7500558.submit parsl.executors.status_handling INFO: Allocated block ID 4712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4712 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4712 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4712.1709793987.750476.submit parsl.executors.status_handling INFO: Allocated block ID 4713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4713 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4713 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4713.1709793987.7509005.submit parsl.executors.status_handling INFO: Allocated block ID 4714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4714 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4714 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4714.1709793987.7513244.submit parsl.executors.status_handling INFO: Allocated block ID 4715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4715 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4715 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4715.1709793987.7517407.submit parsl.executors.status_handling INFO: Allocated block ID 4716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4716 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4716 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4716.1709793987.7521665.submit parsl.executors.status_handling INFO: Allocated block ID 4717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4717 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4717 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4717.1709793987.7525826.submit parsl.executors.status_handling INFO: Allocated block ID 4718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4718 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4718 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4718.1709793987.7530081.submit parsl.executors.status_handling INFO: Allocated block ID 4719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4719 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4719 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4719.1709793987.753429.submit parsl.executors.status_handling INFO: Allocated block ID 4720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4720 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4720 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4720.1709793987.7538595.submit parsl.executors.status_handling INFO: Allocated block ID 4721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4721 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4721 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4721.1709793987.7542887.submit parsl.executors.status_handling INFO: Allocated block ID 4722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4722 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4722 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4722.1709793987.7547092.submit parsl.executors.status_handling INFO: Allocated block ID 4723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4723 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4723 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4723.1709793987.7551467.submit parsl.executors.status_handling INFO: Allocated block ID 4724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4724 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4724 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4724.1709793987.7555637.submit parsl.executors.status_handling INFO: Allocated block ID 4725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4725 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4725 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4725.1709793987.7658002.submit parsl.executors.status_handling INFO: Allocated block ID 4726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4726 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4726 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4726.1709793987.7663136.submit parsl.executors.status_handling INFO: Allocated block ID 4727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4727 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4727 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4727.1709793987.7667453.submit parsl.executors.status_handling INFO: Allocated block ID 4728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4728 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4728 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4728.1709793987.7671819.submit parsl.executors.status_handling INFO: Allocated block ID 4729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4729 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4729 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4729.1709793987.7676117.submit parsl.executors.status_handling INFO: Allocated block ID 4730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4730 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4730 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4730.1709793987.768041.submit parsl.executors.status_handling INFO: Allocated block ID 4731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4731 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4731 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4731.1709793987.768467.submit parsl.executors.status_handling INFO: Allocated block ID 4732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4732 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4732 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4732.1709793987.7689168.submit parsl.executors.status_handling INFO: Allocated block ID 4733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4733 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4733 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4733.1709793987.7693563.submit parsl.executors.status_handling INFO: Allocated block ID 4734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4734 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4734 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4734.1709793987.7697923.submit parsl.executors.status_handling INFO: Allocated block ID 4735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4735 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4735 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4735.1709793987.7702348.submit parsl.executors.status_handling INFO: Allocated block ID 4736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4736 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4736 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4736.1709793987.770669.submit parsl.executors.status_handling INFO: Allocated block ID 4737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4737 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4737 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4737.1709793987.771122.submit parsl.executors.status_handling INFO: Allocated block ID 4738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4738 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4738 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4738.1709793987.77155.submit parsl.executors.status_handling INFO: Allocated block ID 4739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4739 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4739 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4739.1709793987.771978.submit parsl.executors.status_handling INFO: Allocated block ID 4740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4740 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4740 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4740.1709793987.7723987.submit parsl.executors.status_handling INFO: Allocated block ID 4741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4741 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4741 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4741.1709793987.7728226.submit parsl.executors.status_handling INFO: Allocated block ID 4742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4742 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4742 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4742.1709793987.7732446.submit parsl.executors.status_handling INFO: Allocated block ID 4743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4743 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4743 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4743.1709793987.7736638.submit parsl.executors.status_handling INFO: Allocated block ID 4744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4744 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4744 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4744.1709793987.7740946.submit parsl.executors.status_handling INFO: Allocated block ID 4745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4745 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4745 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4745.1709793987.774523.submit parsl.executors.status_handling INFO: Allocated block ID 4746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4746 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4746 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4746.1709793987.7749698.submit parsl.executors.status_handling INFO: Allocated block ID 4747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4747 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4747 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4747.1709793987.7753947.submit parsl.executors.status_handling INFO: Allocated block ID 4748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4748 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4748 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4748.1709793987.775817.submit parsl.executors.status_handling INFO: Allocated block ID 4749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4749 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4749 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4749.1709793987.7762365.submit parsl.executors.status_handling INFO: Allocated block ID 4750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4750 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4750 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4750.1709793987.77666.submit parsl.executors.status_handling INFO: Allocated block ID 4751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4751 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4751 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4751.1709793987.777095.submit parsl.executors.status_handling INFO: Allocated block ID 4752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4752 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4752 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4752.1709793987.7775252.submit parsl.executors.status_handling INFO: Allocated block ID 4753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4753 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4753 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4753.1709793987.78797.submit parsl.executors.status_handling INFO: Allocated block ID 4754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4754 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4754 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4754.1709793987.7884402.submit parsl.executors.status_handling INFO: Allocated block ID 4755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4755 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4755 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4755.1709793987.7888792.submit parsl.executors.status_handling INFO: Allocated block ID 4756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4756 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4756 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4756.1709793987.7893162.submit parsl.executors.status_handling INFO: Allocated block ID 4757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4757 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4757 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4757.1709793987.789749.submit parsl.executors.status_handling INFO: Allocated block ID 4758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4758 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4758 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4758.1709793987.7901852.submit parsl.executors.status_handling INFO: Allocated block ID 4759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4759 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4759 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4759.1709793987.7906084.submit parsl.executors.status_handling INFO: Allocated block ID 4760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4760 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4760 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4760.1709793987.7910352.submit parsl.executors.status_handling INFO: Allocated block ID 4761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4761 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4761 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4761.1709793987.791457.submit parsl.executors.status_handling INFO: Allocated block ID 4762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4762 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4762 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4762.1709793987.7918873.submit parsl.executors.status_handling INFO: Allocated block ID 4763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4763 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4763 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4763.1709793987.7923102.submit parsl.executors.status_handling INFO: Allocated block ID 4764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4764 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4764 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4764.1709793987.7927296.submit parsl.executors.status_handling INFO: Allocated block ID 4765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4765 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4765 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4765.1709793987.793286.submit parsl.executors.status_handling INFO: Allocated block ID 4766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4766 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4766 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4766.1709793987.7937214.submit parsl.executors.status_handling INFO: Allocated block ID 4767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4767 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4767 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4767.1709793987.7941623.submit parsl.executors.status_handling INFO: Allocated block ID 4768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4768 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4768 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4768.1709793987.7946057.submit parsl.executors.status_handling INFO: Allocated block ID 4769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4769 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4769 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4769.1709793987.7950375.submit parsl.executors.status_handling INFO: Allocated block ID 4770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4770 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4770 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4770.1709793987.795462.submit parsl.executors.status_handling INFO: Allocated block ID 4771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4771 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4771 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4771.1709793987.7958884.submit parsl.executors.status_handling INFO: Allocated block ID 4772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4772 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4772 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4772.1709793987.7963188.submit parsl.executors.status_handling INFO: Allocated block ID 4773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4773 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4773 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4773.1709793987.79674.submit parsl.executors.status_handling INFO: Allocated block ID 4774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4774 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4774 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4774.1709793987.797176.submit parsl.executors.status_handling INFO: Allocated block ID 4775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4775 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4775 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4775.1709793987.7975984.submit parsl.executors.status_handling INFO: Allocated block ID 4776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4776 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4776 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4776.1709793987.7980254.submit parsl.executors.status_handling INFO: Allocated block ID 4777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4777 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4777 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4777.1709793987.7984462.submit parsl.executors.status_handling INFO: Allocated block ID 4778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4778 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4778 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4778.1709793987.7988706.submit parsl.executors.status_handling INFO: Allocated block ID 4779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4779 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4779 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4779.1709793987.7992904.submit parsl.executors.status_handling INFO: Allocated block ID 4780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4780 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4780 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4780.1709793987.799718.submit parsl.executors.status_handling INFO: Allocated block ID 4781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4781 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4781 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4781.1709793987.8001552.submit parsl.executors.status_handling INFO: Allocated block ID 4782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4782 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4782 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4782.1709793987.800581.submit parsl.executors.status_handling INFO: Allocated block ID 4783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4783 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4783 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4783.1709793987.8010218.submit parsl.executors.status_handling INFO: Allocated block ID 4784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4784 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4784 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4784.1709793987.801447.submit parsl.executors.status_handling INFO: Allocated block ID 4785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4785 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4785 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4785.1709793987.8037202.submit parsl.executors.status_handling INFO: Allocated block ID 4786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4786 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4786 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4786.1709793987.8042018.submit parsl.executors.status_handling INFO: Allocated block ID 4787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4787 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4787 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4787.1709793987.8046348.submit parsl.executors.status_handling INFO: Allocated block ID 4788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4788 2024-03-06 22:46:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4788 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4788.1709793987.805063.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 736eb51af4dd95a084096c48a9ebe39c with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/371b633c-aee7-447c-a164-fe8f0d13a737_biasIsr_1123832_8.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123832/371b633c-aee7-447c-a164-fe8f0d13a737_biasIsr_1123832_8.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdc7d12da4fab61b177e7079fe681486 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4aec241c-1089-4307-85f8-a8e6f025418d_biasIsr_1123835_8.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123835/4aec241c-1089-4307-85f8-a8e6f025418d_biasIsr_1123835_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1341 parsl.dataflow.strategy DEBUG: Executor multi has 1341 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4789 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4789 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4789.1709793992.675847.submit parsl.executors.status_handling INFO: Allocated block ID 4790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4790 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4790 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4790.1709793992.6779492.submit parsl.executors.status_handling INFO: Allocated block ID 4791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4791 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4791 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4791.1709793992.6789703.submit parsl.executors.status_handling INFO: Allocated block ID 4792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4792 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4792 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4792.1709793992.6798522.submit parsl.executors.status_handling INFO: Allocated block ID 4793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4793 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4793 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4793.1709793992.6807349.submit parsl.executors.status_handling INFO: Allocated block ID 4794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4794 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4794 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4794.1709793992.6816041.submit parsl.executors.status_handling INFO: Allocated block ID 4795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4795 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4795 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4795.1709793992.682474.submit parsl.executors.status_handling INFO: Allocated block ID 4796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4796 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4796 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4796.1709793992.6835208.submit parsl.executors.status_handling INFO: Allocated block ID 4797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4797 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4797 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4797.1709793992.684447.submit parsl.executors.status_handling INFO: Allocated block ID 4798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4798 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4798 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4798.1709793992.685311.submit parsl.executors.status_handling INFO: Allocated block ID 4799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4799 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4799 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4799.1709793992.686177.submit parsl.executors.status_handling INFO: Allocated block ID 4800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4800 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4800 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4800.1709793992.6870427.submit parsl.executors.status_handling INFO: Allocated block ID 4801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4801 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4801 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4801.1709793992.6881337.submit parsl.executors.status_handling INFO: Allocated block ID 4802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4802 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4802 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4802.1709793992.6886876.submit parsl.executors.status_handling INFO: Allocated block ID 4803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4803 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4803 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4803.1709793992.6891158.submit parsl.executors.status_handling INFO: Allocated block ID 4804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4804 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4804 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4804.1709793992.6895351.submit parsl.executors.status_handling INFO: Allocated block ID 4805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4805 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4805 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4805.1709793992.6899652.submit parsl.executors.status_handling INFO: Allocated block ID 4806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4806 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4806 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4806.1709793992.6903841.submit parsl.executors.status_handling INFO: Allocated block ID 4807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4807 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4807 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4807.1709793992.690801.submit parsl.executors.status_handling INFO: Allocated block ID 4808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4808 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4808 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4808.1709793992.6912277.submit parsl.executors.status_handling INFO: Allocated block ID 4809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4809 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4809 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4809.1709793992.6916428.submit parsl.executors.status_handling INFO: Allocated block ID 4810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4810 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4810 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4810.1709793992.6920836.submit parsl.executors.status_handling INFO: Allocated block ID 4811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4811 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4811 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4811.1709793992.6925116.submit parsl.executors.status_handling INFO: Allocated block ID 4812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4812 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4812 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4812.1709793992.6929514.submit parsl.executors.status_handling INFO: Allocated block ID 4813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4813 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4813 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4813.1709793992.693374.submit parsl.executors.status_handling INFO: Allocated block ID 4814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4814 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4814 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4814.1709793992.6942353.submit parsl.executors.status_handling INFO: Allocated block ID 4815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4815 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4815 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4815.1709793992.6946795.submit parsl.executors.status_handling INFO: Allocated block ID 4816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4816 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4816 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4816.1709793992.695133.submit parsl.executors.status_handling INFO: Allocated block ID 4817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4817 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4817 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4817.1709793992.695549.submit parsl.executors.status_handling INFO: Allocated block ID 4818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4818 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4818 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4818.1709793992.6959724.submit parsl.executors.status_handling INFO: Allocated block ID 4819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4819 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4819 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4819.1709793992.6963897.submit parsl.executors.status_handling INFO: Allocated block ID 4820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4820 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4820 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4820.1709793992.6968126.submit parsl.executors.status_handling INFO: Allocated block ID 4821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4821 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4821 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4821.1709793992.6972544.submit parsl.executors.status_handling INFO: Allocated block ID 4822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4822 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4822 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4822.1709793992.6976802.submit parsl.executors.status_handling INFO: Allocated block ID 4823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4823 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4823 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4823.1709793992.6981256.submit parsl.executors.status_handling INFO: Allocated block ID 4824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4824 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4824 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4824.1709793992.698545.submit parsl.executors.status_handling INFO: Allocated block ID 4825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4825 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4825 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4825.1709793992.6989636.submit parsl.executors.status_handling INFO: Allocated block ID 4826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4826 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4826 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4826.1709793992.6993804.submit parsl.executors.status_handling INFO: Allocated block ID 4827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4827 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4827 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4827.1709793992.6997988.submit parsl.executors.status_handling INFO: Allocated block ID 4828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4828 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4828 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4828.1709793992.7002232.submit parsl.executors.status_handling INFO: Allocated block ID 4829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4829 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4829 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4829.1709793992.7006435.submit parsl.executors.status_handling INFO: Allocated block ID 4830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4830 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4830 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4830.1709793992.7010622.submit parsl.executors.status_handling INFO: Allocated block ID 4831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4831 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4831 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4831.1709793992.701479.submit parsl.executors.status_handling INFO: Allocated block ID 4832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4832 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4832 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4832.1709793992.7019038.submit parsl.executors.status_handling INFO: Allocated block ID 4833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4833 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4833 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4833.1709793992.702321.submit parsl.executors.status_handling INFO: Allocated block ID 4834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4834 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4834 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4834.1709793992.7027304.submit parsl.executors.status_handling INFO: Allocated block ID 4835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4835 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4835 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4835.1709793992.7031531.submit parsl.executors.status_handling INFO: Allocated block ID 4836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4836 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4836 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4836.1709793992.7035778.submit parsl.executors.status_handling INFO: Allocated block ID 4837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4837 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4837 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4837.1709793992.704168.submit parsl.executors.status_handling INFO: Allocated block ID 4838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4838 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4838 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4838.1709793992.7046072.submit parsl.executors.status_handling INFO: Allocated block ID 4839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4839 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4839 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4839.1709793992.7050433.submit parsl.executors.status_handling INFO: Allocated block ID 4840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4840 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4840 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4840.1709793992.7054794.submit parsl.executors.status_handling INFO: Allocated block ID 4841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4841 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4841 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4841.1709793992.7059014.submit parsl.executors.status_handling INFO: Allocated block ID 4842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4842 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4842 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4842.1709793992.7063205.submit parsl.executors.status_handling INFO: Allocated block ID 4843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4843 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4843 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4843.1709793992.7067413.submit parsl.executors.status_handling INFO: Allocated block ID 4844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4844 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4844 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4844.1709793992.7071688.submit parsl.executors.status_handling INFO: Allocated block ID 4845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4845 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4845 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4845.1709793992.707582.submit parsl.executors.status_handling INFO: Allocated block ID 4846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4846 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4846 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4846.1709793992.7080019.submit parsl.executors.status_handling INFO: Allocated block ID 4847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4847 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4847 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4847.1709793992.7084136.submit parsl.executors.status_handling INFO: Allocated block ID 4848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4848 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4848 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4848.1709793992.7088294.submit parsl.executors.status_handling INFO: Allocated block ID 4849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4849 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4849 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4849.1709793992.709246.submit parsl.executors.status_handling INFO: Allocated block ID 4850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4850 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4850 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4850.1709793992.70966.submit parsl.executors.status_handling INFO: Allocated block ID 4851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4851 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4851 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4851.1709793992.710089.submit parsl.executors.status_handling INFO: Allocated block ID 4852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4852 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4852 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4852.1709793992.71051.submit parsl.executors.status_handling INFO: Allocated block ID 4853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4853 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4853 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4853.1709793992.710937.submit parsl.executors.status_handling INFO: Allocated block ID 4854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4854 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4854 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4854.1709793992.7113702.submit parsl.executors.status_handling INFO: Allocated block ID 4855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4855 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4855 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4855.1709793992.7117982.submit parsl.executors.status_handling INFO: Allocated block ID 4856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4856 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4856 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4856.1709793992.712235.submit parsl.executors.status_handling INFO: Allocated block ID 4857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4857 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4857 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4857.1709793992.7126734.submit parsl.executors.status_handling INFO: Allocated block ID 4858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4858 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4858 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4858.1709793992.7130983.submit parsl.executors.status_handling INFO: Allocated block ID 4859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4859 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4859 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4859.1709793992.713511.submit parsl.executors.status_handling INFO: Allocated block ID 4860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4860 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4860 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4860.1709793992.7140477.submit parsl.executors.status_handling INFO: Allocated block ID 4861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4861 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4861 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4861.1709793992.7144809.submit parsl.executors.status_handling INFO: Allocated block ID 4862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4862 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4862 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4862.1709793992.7149153.submit parsl.executors.status_handling INFO: Allocated block ID 4863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4863 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4863 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4863.1709793992.7153327.submit parsl.executors.status_handling INFO: Allocated block ID 4864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4864 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4864 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4864.1709793992.7157607.submit parsl.executors.status_handling INFO: Allocated block ID 4865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4865 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4865 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4865.1709793992.7161849.submit parsl.executors.status_handling INFO: Allocated block ID 4866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4866 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4866 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4866.1709793992.7166033.submit parsl.executors.status_handling INFO: Allocated block ID 4867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4867 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4867 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4867.1709793992.7170234.submit parsl.executors.status_handling INFO: Allocated block ID 4868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4868 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4868 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4868.1709793992.7174392.submit parsl.executors.status_handling INFO: Allocated block ID 4869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4869 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4869 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4869.1709793992.717863.submit parsl.executors.status_handling INFO: Allocated block ID 4870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4870 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4870 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4870.1709793992.7182772.submit parsl.executors.status_handling INFO: Allocated block ID 4871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4871 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4871 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4871.1709793992.7186925.submit parsl.executors.status_handling INFO: Allocated block ID 4872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4872 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4872 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4872.1709793992.7191124.submit parsl.executors.status_handling INFO: Allocated block ID 4873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4873 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4873 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4873.1709793992.7195292.submit parsl.executors.status_handling INFO: Allocated block ID 4874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4874 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4874 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4874.1709793992.7199564.submit parsl.executors.status_handling INFO: Allocated block ID 4875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4875 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4875 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4875.1709793992.7203743.submit parsl.executors.status_handling INFO: Allocated block ID 4876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4876 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4876 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4876.1709793992.720794.submit parsl.executors.status_handling INFO: Allocated block ID 4877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4877 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4877 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4877.1709793992.7212226.submit parsl.executors.status_handling INFO: Allocated block ID 4878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4878 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4878 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4878.1709793992.7216392.submit parsl.executors.status_handling INFO: Allocated block ID 4879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4879 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4879 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4879.1709793992.7220628.submit parsl.executors.status_handling INFO: Allocated block ID 4880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4880 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4880 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4880.1709793992.7224813.submit parsl.executors.status_handling INFO: Allocated block ID 4881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4881 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4881 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4881.1709793992.7229002.submit parsl.executors.status_handling INFO: Allocated block ID 4882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4882 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4882 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4882.1709793992.7233138.submit parsl.executors.status_handling INFO: Allocated block ID 4883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4883 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4883 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4883.1709793992.7237284.submit parsl.executors.status_handling INFO: Allocated block ID 4884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4884 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4884 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4884.1709793992.7241561.submit parsl.executors.status_handling INFO: Allocated block ID 4885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4885 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4885 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4885.1709793992.7245693.submit parsl.executors.status_handling INFO: Allocated block ID 4886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4886 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4886 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4886.1709793992.725105.submit parsl.executors.status_handling INFO: Allocated block ID 4887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4887 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4887 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4887.1709793992.7255414.submit parsl.executors.status_handling INFO: Allocated block ID 4888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4888 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4888 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4888.1709793992.7259989.submit parsl.executors.status_handling INFO: Allocated block ID 4889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4889 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4889 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4889.1709793992.7264285.submit parsl.executors.status_handling INFO: Allocated block ID 4890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4890 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4890 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4890.1709793992.7268615.submit parsl.executors.status_handling INFO: Allocated block ID 4891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4891 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4891 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4891.1709793992.7272902.submit parsl.executors.status_handling INFO: Allocated block ID 4892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4892 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4892 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4892.1709793992.727704.submit parsl.executors.status_handling INFO: Allocated block ID 4893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4893 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4893 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4893.1709793992.7281272.submit parsl.executors.status_handling INFO: Allocated block ID 4894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4894 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4894 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4894.1709793992.7285626.submit parsl.executors.status_handling INFO: Allocated block ID 4895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4895 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4895 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4895.1709793992.7289948.submit parsl.executors.status_handling INFO: Allocated block ID 4896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4896 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4896 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4896.1709793992.7294264.submit parsl.executors.status_handling INFO: Allocated block ID 4897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4897 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4897 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4897.1709793992.729856.submit parsl.executors.status_handling INFO: Allocated block ID 4898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4898 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4898 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4898.1709793992.730276.submit parsl.executors.status_handling INFO: Allocated block ID 4899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4899 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4899 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4899.1709793992.7306998.submit parsl.executors.status_handling INFO: Allocated block ID 4900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4900 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4900 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4900.1709793992.7311273.submit parsl.executors.status_handling INFO: Allocated block ID 4901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4901 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4901 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4901.1709793992.7315454.submit parsl.executors.status_handling INFO: Allocated block ID 4902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4902 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4902 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4902.1709793992.7319658.submit parsl.executors.status_handling INFO: Allocated block ID 4903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4903 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4903 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4903.1709793992.7323878.submit parsl.executors.status_handling INFO: Allocated block ID 4904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4904 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4904 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4904.1709793992.7328155.submit parsl.executors.status_handling INFO: Allocated block ID 4905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4905 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4905 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4905.1709793992.7332342.submit parsl.executors.status_handling INFO: Allocated block ID 4906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4906 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4906 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4906.1709793992.7336488.submit parsl.executors.status_handling INFO: Allocated block ID 4907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4907 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4907 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4907.1709793992.7340703.submit parsl.executors.status_handling INFO: Allocated block ID 4908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4908 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4908 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4908.1709793992.734484.submit parsl.executors.status_handling INFO: Allocated block ID 4909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4909 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4909 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4909.1709793992.7349072.submit parsl.executors.status_handling INFO: Allocated block ID 4910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4910 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4910 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4910.1709793992.735333.submit parsl.executors.status_handling INFO: Allocated block ID 4911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4911 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4911 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4911.1709793992.7357466.submit parsl.executors.status_handling INFO: Allocated block ID 4912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4912 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4912 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4912.1709793992.7362938.submit parsl.executors.status_handling INFO: Allocated block ID 4913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4913 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4913 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4913.1709793992.736717.submit parsl.executors.status_handling INFO: Allocated block ID 4914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4914 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4914 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4914.1709793992.7371435.submit parsl.executors.status_handling INFO: Allocated block ID 4915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4915 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4915 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4915.1709793992.7375598.submit parsl.executors.status_handling INFO: Allocated block ID 4916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4916 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4916 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4916.1709793992.737993.submit parsl.executors.status_handling INFO: Allocated block ID 4917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4917 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4917 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4917.1709793992.7384179.submit parsl.executors.status_handling INFO: Allocated block ID 4918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4918 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4918 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4918.1709793992.738838.submit parsl.executors.status_handling INFO: Allocated block ID 4919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4919 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4919 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4919.1709793992.7392645.submit parsl.executors.status_handling INFO: Allocated block ID 4920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4920 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4920 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4920.1709793992.7396815.submit parsl.executors.status_handling INFO: Allocated block ID 4921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4921 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4921 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4921.1709793992.7401457.submit parsl.executors.status_handling INFO: Allocated block ID 4922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4922 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4922 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4922.1709793992.740567.submit parsl.executors.status_handling INFO: Allocated block ID 4923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4923 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4923 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4923.1709793992.7409909.submit parsl.executors.status_handling INFO: Allocated block ID 4924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4924 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4924 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4924.1709793992.7414124.submit parsl.executors.status_handling INFO: Allocated block ID 4925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4925 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4925 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4925.1709793992.7418466.submit parsl.executors.status_handling INFO: Allocated block ID 4926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4926 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4926 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4926.1709793992.7422667.submit parsl.executors.status_handling INFO: Allocated block ID 4927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4927 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4927 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4927.1709793992.7426832.submit parsl.executors.status_handling INFO: Allocated block ID 4928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4928 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4928 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4928.1709793992.7431097.submit parsl.executors.status_handling INFO: Allocated block ID 4929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4929 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4929 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4929.1709793992.7435257.submit parsl.executors.status_handling INFO: Allocated block ID 4930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4930 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4930 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4930.1709793992.743946.submit parsl.executors.status_handling INFO: Allocated block ID 4931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4931 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4931 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4931.1709793992.7443593.submit parsl.executors.status_handling INFO: Allocated block ID 4932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4932 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4932 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4932.1709793992.744781.submit parsl.executors.status_handling INFO: Allocated block ID 4933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4933 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4933 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4933.1709793992.7452016.submit parsl.executors.status_handling INFO: Allocated block ID 4934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4934 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4934 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4934.1709793992.7456229.submit parsl.executors.status_handling INFO: Allocated block ID 4935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4935 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4935 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4935.1709793992.7550344.submit parsl.executors.status_handling INFO: Allocated block ID 4936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4936 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4936 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4936.1709793992.7555397.submit parsl.executors.status_handling INFO: Allocated block ID 4937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4937 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4937 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4937.1709793992.7559757.submit parsl.executors.status_handling INFO: Allocated block ID 4938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4938 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4938 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4938.1709793992.7564998.submit parsl.executors.status_handling INFO: Allocated block ID 4939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4939 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4939 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4939.1709793992.7569315.submit parsl.executors.status_handling INFO: Allocated block ID 4940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4940 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4940 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4940.1709793992.7573576.submit parsl.executors.status_handling INFO: Allocated block ID 4941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4941 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4941 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4941.1709793992.7577746.submit parsl.executors.status_handling INFO: Allocated block ID 4942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4942 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4942 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4942.1709793992.758208.submit parsl.executors.status_handling INFO: Allocated block ID 4943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4943 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4943 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4943.1709793992.758625.submit parsl.executors.status_handling INFO: Allocated block ID 4944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4944 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4944 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4944.1709793992.759046.submit parsl.executors.status_handling INFO: Allocated block ID 4945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4945 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4945 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4945.1709793992.759463.submit parsl.executors.status_handling INFO: Allocated block ID 4946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4946 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4946 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4946.1709793992.7598877.submit parsl.executors.status_handling INFO: Allocated block ID 4947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4947 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4947 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4947.1709793992.7603092.submit parsl.executors.status_handling INFO: Allocated block ID 4948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4948 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4948 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4948.1709793992.7607331.submit parsl.executors.status_handling INFO: Allocated block ID 4949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4949 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4949 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4949.1709793992.7611592.submit parsl.executors.status_handling INFO: Allocated block ID 4950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4950 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4950 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4950.1709793992.761575.submit parsl.executors.status_handling INFO: Allocated block ID 4951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4951 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4951 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4951.1709793992.7619984.submit parsl.executors.status_handling INFO: Allocated block ID 4952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4952 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4952 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4952.1709793992.7624187.submit parsl.executors.status_handling INFO: Allocated block ID 4953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4953 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4953 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4953.1709793992.7628386.submit parsl.executors.status_handling INFO: Allocated block ID 4954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4954 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4954 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4954.1709793992.7632532.submit parsl.executors.status_handling INFO: Allocated block ID 4955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4955 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4955 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4955.1709793992.7636688.submit parsl.executors.status_handling INFO: Allocated block ID 4956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4956 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4956 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4956.1709793992.7640896.submit parsl.executors.status_handling INFO: Allocated block ID 4957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4957 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4957 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4957.1709793992.7645025.submit parsl.executors.status_handling INFO: Allocated block ID 4958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4958 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4958 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4958.1709793992.7739685.submit parsl.executors.status_handling INFO: Allocated block ID 4959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4959 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4959 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4959.1709793992.774439.submit parsl.executors.status_handling INFO: Allocated block ID 4960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4960 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4960 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4960.1709793992.7748752.submit parsl.executors.status_handling INFO: Allocated block ID 4961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4961 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4961 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4961.1709793992.7753422.submit parsl.executors.status_handling INFO: Allocated block ID 4962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4962 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4962 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4962.1709793992.7757664.submit parsl.executors.status_handling INFO: Allocated block ID 4963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4963 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4963 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4963.1709793992.7762084.submit parsl.executors.status_handling INFO: Allocated block ID 4964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4964 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4964 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4964.1709793992.7766364.submit parsl.executors.status_handling INFO: Allocated block ID 4965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4965 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4965 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4965.1709793992.7770646.submit parsl.executors.status_handling INFO: Allocated block ID 4966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4966 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4966 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4966.1709793992.7774816.submit parsl.executors.status_handling INFO: Allocated block ID 4967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4967 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4967 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4967.1709793992.7779093.submit parsl.executors.status_handling INFO: Allocated block ID 4968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4968 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4968 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4968.1709793992.7783272.submit parsl.executors.status_handling INFO: Allocated block ID 4969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4969 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4969 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4969.1709793992.7787433.submit parsl.executors.status_handling INFO: Allocated block ID 4970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4970 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4970 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4970.1709793992.7791653.submit parsl.executors.status_handling INFO: Allocated block ID 4971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4971 2024-03-06 22:46:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4971 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4971.1709793992.779582.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4d3c246896395a93e58dcca61664647 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ce93ad6b-6d67-4c34-b9ce-4e6d20d540b5_biasIsr_1123731_8.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123731/ce93ad6b-6d67-4c34-b9ce-4e6d20d540b5_biasIsr_1123731_8.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77dab41b85833a733f8b7f6e7f89fac7 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/bf504e3e-b733-4eba-9716-31f136e9dff4_biasIsr_1123735_8.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123735/bf504e3e-b733-4eba-9716-31f136e9dff4_biasIsr_1123735_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1339 parsl.dataflow.strategy DEBUG: Executor multi has 1339 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 4972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4972 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4972 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4972.1709793997.669492.submit parsl.executors.status_handling INFO: Allocated block ID 4973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4973 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4973 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4973.1709793997.669966.submit parsl.executors.status_handling INFO: Allocated block ID 4974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4974 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4974 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4974.1709793997.670403.submit parsl.executors.status_handling INFO: Allocated block ID 4975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4975 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4975 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4975.1709793997.6708338.submit parsl.executors.status_handling INFO: Allocated block ID 4976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4976 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4976 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4976.1709793997.6712565.submit parsl.executors.status_handling INFO: Allocated block ID 4977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4977 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4977 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4977.1709793997.6716833.submit parsl.executors.status_handling INFO: Allocated block ID 4978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4978 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4978 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4978.1709793997.6721113.submit parsl.executors.status_handling INFO: Allocated block ID 4979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4979 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4979 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4979.1709793997.6725311.submit parsl.executors.status_handling INFO: Allocated block ID 4980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4980 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4980 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4980.1709793997.6729562.submit parsl.executors.status_handling INFO: Allocated block ID 4981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4981 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4981 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4981.1709793997.6733735.submit parsl.executors.status_handling INFO: Allocated block ID 4982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4982 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4982 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4982.1709793997.6737907.submit parsl.executors.status_handling INFO: Allocated block ID 4983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4983 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4983 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4983.1709793997.6742184.submit parsl.executors.status_handling INFO: Allocated block ID 4984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4984 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4984 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4984.1709793997.6746438.submit parsl.executors.status_handling INFO: Allocated block ID 4985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4985 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4985 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4985.1709793997.6750643.submit parsl.executors.status_handling INFO: Allocated block ID 4986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4986 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4986 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4986.1709793997.6754868.submit parsl.executors.status_handling INFO: Allocated block ID 4987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4987 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4987 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4987.1709793997.6759076.submit parsl.executors.status_handling INFO: Allocated block ID 4988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4988 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4988 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4988.1709793997.6763237.submit parsl.executors.status_handling INFO: Allocated block ID 4989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4989 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4989 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4989.1709793997.6767366.submit parsl.executors.status_handling INFO: Allocated block ID 4990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4990 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4990 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4990.1709793997.6771631.submit parsl.executors.status_handling INFO: Allocated block ID 4991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4991 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4991 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4991.1709793997.67758.submit parsl.executors.status_handling INFO: Allocated block ID 4992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4992 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4992 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4992.1709793997.6780097.submit parsl.executors.status_handling INFO: Allocated block ID 4993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4993 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4993 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4993.1709793997.6784313.submit parsl.executors.status_handling INFO: Allocated block ID 4994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4994 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4994 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4994.1709793997.6788547.submit parsl.executors.status_handling INFO: Allocated block ID 4995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4995 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4995 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4995.1709793997.6792781.submit parsl.executors.status_handling INFO: Allocated block ID 4996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4996 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4996 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4996.1709793997.6796975.submit parsl.executors.status_handling INFO: Allocated block ID 4997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4997 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4997 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4997.1709793997.680246.submit parsl.executors.status_handling INFO: Allocated block ID 4998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4998 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4998 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4998.1709793997.680681.submit parsl.executors.status_handling INFO: Allocated block ID 4999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4999 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4999 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=4999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-4999.1709793997.6811094.submit parsl.executors.status_handling INFO: Allocated block ID 5000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5000 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5000 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5000.1709793997.6815324.submit parsl.executors.status_handling INFO: Allocated block ID 5001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5001 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5001 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5001.1709793997.681964.submit parsl.executors.status_handling INFO: Allocated block ID 5002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5002 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5002 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5002.1709793997.6823864.submit parsl.executors.status_handling INFO: Allocated block ID 5003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5003 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5003 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5003.1709793997.6827984.submit parsl.executors.status_handling INFO: Allocated block ID 5004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5004 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5004 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5004.1709793997.6832206.submit parsl.executors.status_handling INFO: Allocated block ID 5005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5005 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5005 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5005.1709793997.6836343.submit parsl.executors.status_handling INFO: Allocated block ID 5006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5006 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5006 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5006.1709793997.6840613.submit parsl.executors.status_handling INFO: Allocated block ID 5007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5007 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5007 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5007.1709793997.6844842.submit parsl.executors.status_handling INFO: Allocated block ID 5008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5008 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5008 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5008.1709793997.6849148.submit parsl.executors.status_handling INFO: Allocated block ID 5009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5009 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5009 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5009.1709793997.6853342.submit parsl.executors.status_handling INFO: Allocated block ID 5010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5010 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5010 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5010.1709793997.6857653.submit parsl.executors.status_handling INFO: Allocated block ID 5011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5011 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5011 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5011.1709793997.6861992.submit parsl.executors.status_handling INFO: Allocated block ID 5012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5012 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5012 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5012.1709793997.6866224.submit parsl.executors.status_handling INFO: Allocated block ID 5013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5013 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5013 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5013.1709793997.687047.submit parsl.executors.status_handling INFO: Allocated block ID 5014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5014 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5014 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5014.1709793997.687467.submit parsl.executors.status_handling INFO: Allocated block ID 5015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5015 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5015 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5015.1709793997.687889.submit parsl.executors.status_handling INFO: Allocated block ID 5016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5016 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5016 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5016.1709793997.6883035.submit parsl.executors.status_handling INFO: Allocated block ID 5017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5017 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5017 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5017.1709793997.6887162.submit parsl.executors.status_handling INFO: Allocated block ID 5018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5018 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5018 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5018.1709793997.7003217.submit parsl.executors.status_handling INFO: Allocated block ID 5019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5019 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5019 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5019.1709793997.7007563.submit parsl.executors.status_handling INFO: Allocated block ID 5020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5020 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5020 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5020.1709793997.7011912.submit parsl.executors.status_handling INFO: Allocated block ID 5021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5021 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5021 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5021.1709793997.701611.submit parsl.executors.status_handling INFO: Allocated block ID 5022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5022 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5022 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5022.1709793997.7020445.submit parsl.executors.status_handling INFO: Allocated block ID 5023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5023 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5023 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5023.1709793997.7024696.submit parsl.executors.status_handling INFO: Allocated block ID 5024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5024 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5024 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5024.1709793997.7028944.submit parsl.executors.status_handling INFO: Allocated block ID 5025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5025 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5025 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5025.1709793997.70331.submit parsl.executors.status_handling INFO: Allocated block ID 5026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5026 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5026 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5026.1709793997.7037258.submit parsl.executors.status_handling INFO: Allocated block ID 5027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5027 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5027 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5027.1709793997.7042148.submit parsl.executors.status_handling INFO: Allocated block ID 5028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5028 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5028 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5028.1709793997.704634.submit parsl.executors.status_handling INFO: Allocated block ID 5029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5029 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5029 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5029.1709793997.7050512.submit parsl.executors.status_handling INFO: Allocated block ID 5030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5030 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5030 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5030.1709793997.7054644.submit parsl.executors.status_handling INFO: Allocated block ID 5031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5031 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5031 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5031.1709793997.7058911.submit parsl.executors.status_handling INFO: Allocated block ID 5032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5032 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5032 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5032.1709793997.7063177.submit parsl.executors.status_handling INFO: Allocated block ID 5033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5033 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5033 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5033.1709793997.70674.submit parsl.executors.status_handling INFO: Allocated block ID 5034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5034 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5034 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5034.1709793997.7071788.submit parsl.executors.status_handling INFO: Allocated block ID 5035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5035 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5035 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5035.1709793997.7076054.submit parsl.executors.status_handling INFO: Allocated block ID 5036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5036 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5036 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5036.1709793997.708035.submit parsl.executors.status_handling INFO: Allocated block ID 5037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5037 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5037 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5037.1709793997.7084656.submit parsl.executors.status_handling INFO: Allocated block ID 5038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5038 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5038 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5038.1709793997.7189865.submit parsl.executors.status_handling INFO: Allocated block ID 5039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5039 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5039 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5039.1709793997.7194672.submit parsl.executors.status_handling INFO: Allocated block ID 5040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5040 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5040 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5040.1709793997.7199264.submit parsl.executors.status_handling INFO: Allocated block ID 5041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5041 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5041 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5041.1709793997.7205048.submit parsl.executors.status_handling INFO: Allocated block ID 5042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5042 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5042 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5042.1709793997.720956.submit parsl.executors.status_handling INFO: Allocated block ID 5043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5043 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5043 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5043.1709793997.7213998.submit parsl.executors.status_handling INFO: Allocated block ID 5044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5044 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5044 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5044.1709793997.721833.submit parsl.executors.status_handling INFO: Allocated block ID 5045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5045 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5045 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5045.1709793997.7222564.submit parsl.executors.status_handling INFO: Allocated block ID 5046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5046 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5046 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5046.1709793997.722692.submit parsl.executors.status_handling INFO: Allocated block ID 5047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5047 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5047 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5047.1709793997.723126.submit parsl.executors.status_handling INFO: Allocated block ID 5048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5048 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5048 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5048.1709793997.7235491.submit parsl.executors.status_handling INFO: Allocated block ID 5049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5049 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5049 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5049.1709793997.723978.submit parsl.executors.status_handling INFO: Allocated block ID 5050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5050 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5050 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5050.1709793997.7244058.submit parsl.executors.status_handling INFO: Allocated block ID 5051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5051 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5051 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5051.1709793997.7248323.submit parsl.executors.status_handling INFO: Allocated block ID 5052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5052 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5052 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5052.1709793997.7252517.submit parsl.executors.status_handling INFO: Allocated block ID 5053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5053 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5053 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5053.1709793997.725667.submit parsl.executors.status_handling INFO: Allocated block ID 5054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5054 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5054 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5054.1709793997.7261093.submit parsl.executors.status_handling INFO: Allocated block ID 5055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5055 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5055 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5055.1709793997.7265348.submit parsl.executors.status_handling INFO: Allocated block ID 5056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5056 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5056 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5056.1709793997.726968.submit parsl.executors.status_handling INFO: Allocated block ID 5057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5057 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5057 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5057.1709793997.7273939.submit parsl.executors.status_handling INFO: Allocated block ID 5058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5058 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5058 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5058.1709793997.7278373.submit parsl.executors.status_handling INFO: Allocated block ID 5059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5059 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5059 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5059.1709793997.7282732.submit parsl.executors.status_handling INFO: Allocated block ID 5060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5060 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5060 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5060.1709793997.7286892.submit parsl.executors.status_handling INFO: Allocated block ID 5061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5061 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5061 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5061.1709793997.741634.submit parsl.executors.status_handling INFO: Allocated block ID 5062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5062 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5062 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5062.1709793997.742091.submit parsl.executors.status_handling INFO: Allocated block ID 5063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5063 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5063 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5063.1709793997.7425194.submit parsl.executors.status_handling INFO: Allocated block ID 5064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5064 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5064 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5064.1709793997.7429543.submit parsl.executors.status_handling INFO: Allocated block ID 5065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5065 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5065 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5065.1709793997.7433746.submit parsl.executors.status_handling INFO: Allocated block ID 5066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5066 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5066 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5066.1709793997.743793.submit parsl.executors.status_handling INFO: Allocated block ID 5067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5067 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5067 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5067.1709793997.7442217.submit parsl.executors.status_handling INFO: Allocated block ID 5068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5068 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5068 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5068.1709793997.7446427.submit parsl.executors.status_handling INFO: Allocated block ID 5069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5069 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5069 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5069.1709793997.7450707.submit parsl.executors.status_handling INFO: Allocated block ID 5070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5070 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5070 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5070.1709793997.7454984.submit parsl.executors.status_handling INFO: Allocated block ID 5071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5071 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5071 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5071.1709793997.745923.submit parsl.executors.status_handling INFO: Allocated block ID 5072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5072 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5072 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5072.1709793997.7463446.submit parsl.executors.status_handling INFO: Allocated block ID 5073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5073 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5073 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5073.1709793997.746767.submit parsl.executors.status_handling INFO: Allocated block ID 5074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5074 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5074 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5074.1709793997.7472036.submit parsl.executors.status_handling INFO: Allocated block ID 5075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5075 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5075 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5075.1709793997.7476263.submit parsl.executors.status_handling INFO: Allocated block ID 5076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5076 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5076 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5076.1709793997.748057.submit parsl.executors.status_handling INFO: Allocated block ID 5077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5077 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5077 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5077.1709793997.7484746.submit parsl.executors.status_handling INFO: Allocated block ID 5078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5078 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5078 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5078.1709793997.7488902.submit parsl.executors.status_handling INFO: Allocated block ID 5079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5079 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5079 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5079.1709793997.7493012.submit parsl.executors.status_handling INFO: Allocated block ID 5080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5080 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5080 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5080.1709793997.7497182.submit parsl.executors.status_handling INFO: Allocated block ID 5081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5081 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5081 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5081.1709793997.7631686.submit parsl.executors.status_handling INFO: Allocated block ID 5082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5082 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5082 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5082.1709793997.7636087.submit parsl.executors.status_handling INFO: Allocated block ID 5083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5083 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5083 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5083.1709793997.7640414.submit parsl.executors.status_handling INFO: Allocated block ID 5084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5084 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5084 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5084.1709793997.7644618.submit parsl.executors.status_handling INFO: Allocated block ID 5085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5085 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5085 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5085.1709793997.7648847.submit parsl.executors.status_handling INFO: Allocated block ID 5086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5086 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5086 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5086.1709793997.7653103.submit parsl.executors.status_handling INFO: Allocated block ID 5087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5087 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5087 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5087.1709793997.7657256.submit parsl.executors.status_handling INFO: Allocated block ID 5088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5088 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5088 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5088.1709793997.7661512.submit parsl.executors.status_handling INFO: Allocated block ID 5089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5089 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5089 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5089.1709793997.766571.submit parsl.executors.status_handling INFO: Allocated block ID 5090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5090 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5090 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5090.1709793997.766989.submit parsl.executors.status_handling INFO: Allocated block ID 5091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5091 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5091 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5091.1709793997.7674303.submit parsl.executors.status_handling INFO: Allocated block ID 5092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5092 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5092 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5092.1709793997.7678685.submit parsl.executors.status_handling INFO: Allocated block ID 5093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5093 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5093 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5093.1709793997.7682974.submit parsl.executors.status_handling INFO: Allocated block ID 5094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5094 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5094 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5094.1709793997.7687309.submit parsl.executors.status_handling INFO: Allocated block ID 5095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5095 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5095 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5095.1709793997.769157.submit parsl.executors.status_handling INFO: Allocated block ID 5096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5096 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5096 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5096.1709793997.769573.submit parsl.executors.status_handling INFO: Allocated block ID 5097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5097 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5097 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5097.1709793997.7700014.submit parsl.executors.status_handling INFO: Allocated block ID 5098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5098 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5098 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5098.1709793997.784511.submit parsl.executors.status_handling INFO: Allocated block ID 5099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5099 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5099 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5099.1709793997.784977.submit parsl.executors.status_handling INFO: Allocated block ID 5100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5100 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5100 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5100.1709793997.7854054.submit parsl.executors.status_handling INFO: Allocated block ID 5101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5101 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5101 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5101.1709793997.785833.submit parsl.executors.status_handling INFO: Allocated block ID 5102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5102 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5102 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5102.1709793997.7862525.submit parsl.executors.status_handling INFO: Allocated block ID 5103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5103 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5103 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5103.1709793997.7866724.submit parsl.executors.status_handling INFO: Allocated block ID 5104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5104 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5104 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5104.1709793997.7871044.submit parsl.executors.status_handling INFO: Allocated block ID 5105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5105 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5105 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5105.1709793997.7875183.submit parsl.executors.status_handling INFO: Allocated block ID 5106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5106 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5106 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5106.1709793997.7879498.submit parsl.executors.status_handling INFO: Allocated block ID 5107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5107 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5107 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5107.1709793997.7883716.submit parsl.executors.status_handling INFO: Allocated block ID 5108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5108 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5108 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5108.1709793997.7887912.submit parsl.executors.status_handling INFO: Allocated block ID 5109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5109 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5109 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5109.1709793997.789222.submit parsl.executors.status_handling INFO: Allocated block ID 5110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5110 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5110 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5110.1709793997.7896476.submit parsl.executors.status_handling INFO: Allocated block ID 5111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5111 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5111 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5111.1709793997.7900784.submit parsl.executors.status_handling INFO: Allocated block ID 5112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5112 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5112 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5112.1709793997.8057134.submit parsl.executors.status_handling INFO: Allocated block ID 5113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5113 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5113 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5113.1709793997.806211.submit parsl.executors.status_handling INFO: Allocated block ID 5114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5114 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5114 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5114.1709793997.8066432.submit parsl.executors.status_handling INFO: Allocated block ID 5115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5115 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5115 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5115.1709793997.8070757.submit parsl.executors.status_handling INFO: Allocated block ID 5116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5116 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5116 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5116.1709793997.8075075.submit parsl.executors.status_handling INFO: Allocated block ID 5117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5117 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5117 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5117.1709793997.807933.submit parsl.executors.status_handling INFO: Allocated block ID 5118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5118 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5118 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5118.1709793997.8083625.submit parsl.executors.status_handling INFO: Allocated block ID 5119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5119 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5119 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5119.1709793997.8087835.submit parsl.executors.status_handling INFO: Allocated block ID 5120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5120 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5120 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5120.1709793997.8092139.submit parsl.executors.status_handling INFO: Allocated block ID 5121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5121 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5121 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5121.1709793997.8096344.submit parsl.executors.status_handling INFO: Allocated block ID 5122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5122 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5122 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5122.1709793997.8100727.submit parsl.executors.status_handling INFO: Allocated block ID 5123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5123 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5123 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5123.1709793997.8105.submit parsl.executors.status_handling INFO: Allocated block ID 5124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5124 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5124 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5124.1709793997.8230917.submit parsl.executors.status_handling INFO: Allocated block ID 5125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5125 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5125 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5125.1709793997.8235412.submit parsl.executors.status_handling INFO: Allocated block ID 5126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5126 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5126 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5126.1709793997.8239796.submit parsl.executors.status_handling INFO: Allocated block ID 5127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5127 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5127 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5127.1709793997.8244143.submit parsl.executors.status_handling INFO: Allocated block ID 5128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5128 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5128 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5128.1709793997.8248522.submit parsl.executors.status_handling INFO: Allocated block ID 5129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5129 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5129 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5129.1709793997.825277.submit parsl.executors.status_handling INFO: Allocated block ID 5130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5130 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5130 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5130.1709793997.8257058.submit parsl.executors.status_handling INFO: Allocated block ID 5131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5131 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5131 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5131.1709793997.8261397.submit parsl.executors.status_handling INFO: Allocated block ID 5132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5132 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5132 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5132.1709793997.8265672.submit parsl.executors.status_handling INFO: Allocated block ID 5133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5133 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5133 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5133.1709793997.827006.submit parsl.executors.status_handling INFO: Allocated block ID 5134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5134 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5134 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5134.1709793997.8274357.submit parsl.executors.status_handling INFO: Allocated block ID 5135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5135 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5135 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5135.1709793997.8278685.submit parsl.executors.status_handling INFO: Allocated block ID 5136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5136 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5136 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5136.1709793997.828298.submit parsl.executors.status_handling INFO: Allocated block ID 5137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5137 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5137 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5137.1709793997.8287225.submit parsl.executors.status_handling INFO: Allocated block ID 5138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5138 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5138 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5138.1709793997.8291829.submit parsl.executors.status_handling INFO: Allocated block ID 5139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5139 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5139 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5139.1709793997.829604.submit parsl.executors.status_handling INFO: Allocated block ID 5140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5140 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5140 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5140.1709793997.8300316.submit parsl.executors.status_handling INFO: Allocated block ID 5141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5141 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5141 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5141.1709793997.8304546.submit parsl.executors.status_handling INFO: Allocated block ID 5142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5142 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5142 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5142.1709793997.850022.submit parsl.executors.status_handling INFO: Allocated block ID 5143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5143 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5143 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5143.1709793997.850491.submit parsl.executors.status_handling INFO: Allocated block ID 5144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5144 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5144 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5144.1709793997.8509352.submit parsl.executors.status_handling INFO: Allocated block ID 5145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5145 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5145 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5145.1709793997.8513632.submit parsl.executors.status_handling INFO: Allocated block ID 5146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5146 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5146 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5146.1709793997.8517885.submit parsl.executors.status_handling INFO: Allocated block ID 5147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5147 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5147 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5147.1709793997.8522189.submit parsl.executors.status_handling INFO: Allocated block ID 5148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5148 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5148 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5148.1709793997.8526344.submit parsl.executors.status_handling INFO: Allocated block ID 5149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5149 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5149 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5149.1709793997.8530538.submit parsl.executors.status_handling INFO: Allocated block ID 5150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5150 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5150 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5150.1709793997.853474.submit parsl.executors.status_handling INFO: Allocated block ID 5151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5151 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5151 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5151.1709793997.8538976.submit parsl.executors.status_handling INFO: Allocated block ID 5152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5152 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5152 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5152.1709793997.8543234.submit parsl.executors.status_handling INFO: Allocated block ID 5153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5153 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5153 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5153.1709793997.8547418.submit parsl.executors.status_handling INFO: Allocated block ID 5154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5154 2024-03-06 22:46:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5154 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5154.1709793997.855177.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1339 parsl.dataflow.strategy DEBUG: Executor multi has 1339 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 5155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5155 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5155 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5155.1709794002.6726558.submit parsl.executors.status_handling INFO: Allocated block ID 5156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5156 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5156 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5156.1709794002.673176.submit parsl.executors.status_handling INFO: Allocated block ID 5157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5157 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5157 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5157.1709794002.6736214.submit parsl.executors.status_handling INFO: Allocated block ID 5158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5158 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5158 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5158.1709794002.6740544.submit parsl.executors.status_handling INFO: Allocated block ID 5159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5159 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5159 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5159.1709794002.6744797.submit parsl.executors.status_handling INFO: Allocated block ID 5160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5160 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5160 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5160.1709794002.674907.submit parsl.executors.status_handling INFO: Allocated block ID 5161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5161 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5161 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5161.1709794002.6753302.submit parsl.executors.status_handling INFO: Allocated block ID 5162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5162 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5162 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5162.1709794002.6757567.submit parsl.executors.status_handling INFO: Allocated block ID 5163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5163 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5163 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5163.1709794002.676182.submit parsl.executors.status_handling INFO: Allocated block ID 5164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5164 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5164 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5164.1709794002.6766012.submit parsl.executors.status_handling INFO: Allocated block ID 5165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5165 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5165 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5165.1709794002.6770244.submit parsl.executors.status_handling INFO: Allocated block ID 5166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5166 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5166 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5166.1709794002.677448.submit parsl.executors.status_handling INFO: Allocated block ID 5167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5167 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5167 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5167.1709794002.677889.submit parsl.executors.status_handling INFO: Allocated block ID 5168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5168 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5168 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5168.1709794002.678322.submit parsl.executors.status_handling INFO: Allocated block ID 5169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5169 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5169 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5169.1709794002.678754.submit parsl.executors.status_handling INFO: Allocated block ID 5170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5170 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5170 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5170.1709794002.679202.submit parsl.executors.status_handling INFO: Allocated block ID 5171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5171 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5171 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5171.1709794002.6796196.submit parsl.executors.status_handling INFO: Allocated block ID 5172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5172 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5172 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5172.1709794002.680056.submit parsl.executors.status_handling INFO: Allocated block ID 5173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5173 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5173 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5173.1709794002.6804743.submit parsl.executors.status_handling INFO: Allocated block ID 5174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5174 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5174 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5174.1709794002.6808996.submit parsl.executors.status_handling INFO: Allocated block ID 5175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5175 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5175 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5175.1709794002.6813333.submit parsl.executors.status_handling INFO: Allocated block ID 5176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5176 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5176 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5176.1709794002.681758.submit parsl.executors.status_handling INFO: Allocated block ID 5177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5177 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5177 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5177.1709794002.6825526.submit parsl.executors.status_handling INFO: Allocated block ID 5178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5178 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5178 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5178.1709794002.68302.submit parsl.executors.status_handling INFO: Allocated block ID 5179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5179 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5179 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5179.1709794002.6834474.submit parsl.executors.status_handling INFO: Allocated block ID 5180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5180 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5180 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5180.1709794002.6847217.submit parsl.executors.status_handling INFO: Allocated block ID 5181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5181 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5181 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5181.1709794002.6852982.submit parsl.executors.status_handling INFO: Allocated block ID 5182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5182 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5182 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5182.1709794002.6858416.submit parsl.executors.status_handling INFO: Allocated block ID 5183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5183 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5183 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5183.1709794002.6863647.submit parsl.executors.status_handling INFO: Allocated block ID 5184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5184 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5184 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5184.1709794002.6868873.submit parsl.executors.status_handling INFO: Allocated block ID 5185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5185 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5185 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5185.1709794002.6874003.submit parsl.executors.status_handling INFO: Allocated block ID 5186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5186 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5186 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5186.1709794002.6879241.submit parsl.executors.status_handling INFO: Allocated block ID 5187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5187 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5187 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5187.1709794002.688448.submit parsl.executors.status_handling INFO: Allocated block ID 5188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5188 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5188 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5188.1709794002.6889727.submit parsl.executors.status_handling INFO: Allocated block ID 5189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5189 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5189 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5189.1709794002.6894877.submit parsl.executors.status_handling INFO: Allocated block ID 5190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5190 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5190 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5190.1709794002.6900072.submit parsl.executors.status_handling INFO: Allocated block ID 5191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5191 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5191 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5191.1709794002.6905184.submit parsl.executors.status_handling INFO: Allocated block ID 5192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5192 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5192 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5192.1709794002.6917691.submit parsl.executors.status_handling INFO: Allocated block ID 5193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5193 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5193 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5193.1709794002.692352.submit parsl.executors.status_handling INFO: Allocated block ID 5194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5194 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5194 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5194.1709794002.6928854.submit parsl.executors.status_handling INFO: Allocated block ID 5195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5195 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5195 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5195.1709794002.693402.submit parsl.executors.status_handling INFO: Allocated block ID 5196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5196 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5196 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5196.1709794002.6939197.submit parsl.executors.status_handling INFO: Allocated block ID 5197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5197 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5197 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5197.1709794002.69444.submit parsl.executors.status_handling INFO: Allocated block ID 5198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5198 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5198 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5198.1709794002.6949677.submit parsl.executors.status_handling INFO: Allocated block ID 5199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5199 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5199 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5199.1709794002.6954794.submit parsl.executors.status_handling INFO: Allocated block ID 5200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5200 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5200 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5200.1709794002.6960008.submit parsl.executors.status_handling INFO: Allocated block ID 5201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5201 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5201 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5201.1709794002.696514.submit parsl.executors.status_handling INFO: Allocated block ID 5202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5202 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5202 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5202.1709794002.6970348.submit parsl.executors.status_handling INFO: Allocated block ID 5203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5203 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5203 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5203.1709794002.6975484.submit parsl.executors.status_handling INFO: Allocated block ID 5204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5204 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5204 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5204.1709794002.6980705.submit parsl.executors.status_handling INFO: Allocated block ID 5205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5205 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5205 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5205.1709794002.6985948.submit parsl.executors.status_handling INFO: Allocated block ID 5206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5206 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5206 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5206.1709794002.6991184.submit parsl.executors.status_handling INFO: Allocated block ID 5207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5207 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5207 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5207.1709794002.6996334.submit parsl.executors.status_handling INFO: Allocated block ID 5208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5208 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5208 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5208.1709794002.7003675.submit parsl.executors.status_handling INFO: Allocated block ID 5209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5209 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5209 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5209.1709794002.700901.submit parsl.executors.status_handling INFO: Allocated block ID 5210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5210 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5210 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5210.1709794002.701422.submit parsl.executors.status_handling INFO: Allocated block ID 5211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5211 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5211 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5211.1709794002.7019818.submit parsl.executors.status_handling INFO: Allocated block ID 5212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5212 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5212 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5212.1709794002.7025027.submit parsl.executors.status_handling INFO: Allocated block ID 5213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5213 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5213 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5213.1709794002.7030199.submit parsl.executors.status_handling INFO: Allocated block ID 5214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5214 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5214 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5214.1709794002.7035375.submit parsl.executors.status_handling INFO: Allocated block ID 5215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5215 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5215 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5215.1709794002.7040625.submit parsl.executors.status_handling INFO: Allocated block ID 5216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5216 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5216 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5216.1709794002.7045836.submit parsl.executors.status_handling INFO: Allocated block ID 5217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5217 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5217 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5217.1709794002.70522.submit parsl.executors.status_handling INFO: Allocated block ID 5218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5218 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5218 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5218.1709794002.705757.submit parsl.executors.status_handling INFO: Allocated block ID 5219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5219 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5219 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5219.1709794002.7062805.submit parsl.executors.status_handling INFO: Allocated block ID 5220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5220 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5220 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5220.1709794002.706797.submit parsl.executors.status_handling INFO: Allocated block ID 5221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5221 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5221 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5221.1709794002.7073162.submit parsl.executors.status_handling INFO: Allocated block ID 5222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5222 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5222 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5222.1709794002.7078419.submit parsl.executors.status_handling INFO: Allocated block ID 5223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5223 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5223 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5223.1709794002.7083564.submit parsl.executors.status_handling INFO: Allocated block ID 5224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5224 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5224 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5224.1709794002.7088847.submit parsl.executors.status_handling INFO: Allocated block ID 5225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5225 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5225 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5225.1709794002.7094016.submit parsl.executors.status_handling INFO: Allocated block ID 5226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5226 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5226 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5226.1709794002.709924.submit parsl.executors.status_handling INFO: Allocated block ID 5227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5227 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5227 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5227.1709794002.710447.submit parsl.executors.status_handling INFO: Allocated block ID 5228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5228 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5228 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5228.1709794002.7119472.submit parsl.executors.status_handling INFO: Allocated block ID 5229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5229 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5229 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5229.1709794002.7125068.submit parsl.executors.status_handling INFO: Allocated block ID 5230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5230 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5230 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5230.1709794002.7131584.submit parsl.executors.status_handling INFO: Allocated block ID 5231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5231 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5231 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5231.1709794002.7136967.submit parsl.executors.status_handling INFO: Allocated block ID 5232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5232 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5232 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5232.1709794002.714231.submit parsl.executors.status_handling INFO: Allocated block ID 5233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5233 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5233 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5233.1709794002.714747.submit parsl.executors.status_handling INFO: Allocated block ID 5234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5234 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5234 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5234.1709794002.715272.submit parsl.executors.status_handling INFO: Allocated block ID 5235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5235 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5235 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5235.1709794002.7157817.submit parsl.executors.status_handling INFO: Allocated block ID 5236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5236 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5236 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5236.1709794002.7163038.submit parsl.executors.status_handling INFO: Allocated block ID 5237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5237 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5237 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5237.1709794002.7168193.submit parsl.executors.status_handling INFO: Allocated block ID 5238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5238 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5238 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5238.1709794002.7173328.submit parsl.executors.status_handling INFO: Allocated block ID 5239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5239 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5239 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5239.1709794002.7185538.submit parsl.executors.status_handling INFO: Allocated block ID 5240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5240 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5240 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5240.1709794002.7191296.submit parsl.executors.status_handling INFO: Allocated block ID 5241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5241 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5241 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5241.1709794002.719661.submit parsl.executors.status_handling INFO: Allocated block ID 5242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5242 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5242 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5242.1709794002.7202778.submit parsl.executors.status_handling INFO: Allocated block ID 5243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5243 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5243 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5243.1709794002.7207992.submit parsl.executors.status_handling INFO: Allocated block ID 5244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5244 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5244 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5244.1709794002.721327.submit parsl.executors.status_handling INFO: Allocated block ID 5245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5245 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5245 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5245.1709794002.7218792.submit parsl.executors.status_handling INFO: Allocated block ID 5246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5246 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5246 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5246.1709794002.7224088.submit parsl.executors.status_handling INFO: Allocated block ID 5247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5247 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5247 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5247.1709794002.72294.submit parsl.executors.status_handling INFO: Allocated block ID 5248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5248 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5248 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5248.1709794002.7234547.submit parsl.executors.status_handling INFO: Allocated block ID 5249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5249 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5249 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5249.1709794002.7249506.submit parsl.executors.status_handling INFO: Allocated block ID 5250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5250 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5250 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5250.1709794002.7255092.submit parsl.executors.status_handling INFO: Allocated block ID 5251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5251 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5251 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5251.1709794002.7260404.submit parsl.executors.status_handling INFO: Allocated block ID 5252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5252 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5252 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5252.1709794002.7265675.submit parsl.executors.status_handling INFO: Allocated block ID 5253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5253 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5253 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5253.1709794002.727099.submit parsl.executors.status_handling INFO: Allocated block ID 5254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5254 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5254 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5254.1709794002.7276168.submit parsl.executors.status_handling INFO: Allocated block ID 5255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5255 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5255 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5255.1709794002.7281392.submit parsl.executors.status_handling INFO: Allocated block ID 5256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5256 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5256 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5256.1709794002.728653.submit parsl.executors.status_handling INFO: Allocated block ID 5257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5257 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5257 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5257.1709794002.7291725.submit parsl.executors.status_handling INFO: Allocated block ID 5258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5258 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5258 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5258.1709794002.7296886.submit parsl.executors.status_handling INFO: Allocated block ID 5259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5259 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5259 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5259.1709794002.7302206.submit parsl.executors.status_handling INFO: Allocated block ID 5260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5260 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5260 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5260.1709794002.730737.submit parsl.executors.status_handling INFO: Allocated block ID 5261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5261 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5261 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5261.1709794002.7312713.submit parsl.executors.status_handling INFO: Allocated block ID 5262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5262 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5262 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5262.1709794002.7317886.submit parsl.executors.status_handling INFO: Allocated block ID 5263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5263 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5263 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5263.1709794002.733084.submit parsl.executors.status_handling INFO: Allocated block ID 5264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5264 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5264 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5264.1709794002.7336197.submit parsl.executors.status_handling INFO: Allocated block ID 5265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5265 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5265 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5265.1709794002.7341614.submit parsl.executors.status_handling INFO: Allocated block ID 5266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5266 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5266 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5266.1709794002.7346919.submit parsl.executors.status_handling INFO: Allocated block ID 5267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5267 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5267 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5267.1709794002.7352247.submit parsl.executors.status_handling INFO: Allocated block ID 5268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5268 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5268 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5268.1709794002.7357407.submit parsl.executors.status_handling INFO: Allocated block ID 5269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5269 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5269 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5269.1709794002.7362647.submit parsl.executors.status_handling INFO: Allocated block ID 5270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5270 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5270 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5270.1709794002.736785.submit parsl.executors.status_handling INFO: Allocated block ID 5271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5271 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5271 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5271.1709794002.7383118.submit parsl.executors.status_handling INFO: Allocated block ID 5272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5272 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5272 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5272.1709794002.7388923.submit parsl.executors.status_handling INFO: Allocated block ID 5273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5273 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5273 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5273.1709794002.7394435.submit parsl.executors.status_handling INFO: Allocated block ID 5274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5274 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5274 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5274.1709794002.7406921.submit parsl.executors.status_handling INFO: Allocated block ID 5275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5275 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5275 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5275.1709794002.741249.submit parsl.executors.status_handling INFO: Allocated block ID 5276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5276 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5276 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5276.1709794002.7417767.submit parsl.executors.status_handling INFO: Allocated block ID 5277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5277 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5277 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5277.1709794002.7423131.submit parsl.executors.status_handling INFO: Allocated block ID 5278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5278 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5278 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5278.1709794002.7434952.submit parsl.executors.status_handling INFO: Allocated block ID 5279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5279 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5279 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5279.1709794002.7441092.submit parsl.executors.status_handling INFO: Allocated block ID 5280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5280 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5280 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5280.1709794002.7446644.submit parsl.executors.status_handling INFO: Allocated block ID 5281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5281 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5281 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5281.1709794002.7452185.submit parsl.executors.status_handling INFO: Allocated block ID 5282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5282 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5282 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5282.1709794002.7457824.submit parsl.executors.status_handling INFO: Allocated block ID 5283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5283 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5283 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5283.1709794002.747116.submit parsl.executors.status_handling INFO: Allocated block ID 5284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5284 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5284 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5284.1709794002.7476494.submit parsl.executors.status_handling INFO: Allocated block ID 5285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5285 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5285 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5285.1709794002.7491138.submit parsl.executors.status_handling INFO: Allocated block ID 5286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5286 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5286 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5286.1709794002.749655.submit parsl.executors.status_handling INFO: Allocated block ID 5287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5287 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5287 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5287.1709794002.7501838.submit parsl.executors.status_handling INFO: Allocated block ID 5288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5288 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5288 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5288.1709794002.7507074.submit parsl.executors.status_handling INFO: Allocated block ID 5289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5289 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5289 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5289.1709794002.7519262.submit parsl.executors.status_handling INFO: Allocated block ID 5290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5290 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5290 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5290.1709794002.7524905.submit parsl.executors.status_handling INFO: Allocated block ID 5291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5291 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5291 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5291.1709794002.7530534.submit parsl.executors.status_handling INFO: Allocated block ID 5292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5292 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5292 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5292.1709794002.7536132.submit parsl.executors.status_handling INFO: Allocated block ID 5293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5293 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5293 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5293.1709794002.7542477.submit parsl.executors.status_handling INFO: Allocated block ID 5294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5294 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5294 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5294.1709794002.7547982.submit parsl.executors.status_handling INFO: Allocated block ID 5295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5295 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5295 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5295.1709794002.7553575.submit parsl.executors.status_handling INFO: Allocated block ID 5296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5296 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5296 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5296.1709794002.7559137.submit parsl.executors.status_handling INFO: Allocated block ID 5297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5297 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5297 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5297.1709794002.7564383.submit parsl.executors.status_handling INFO: Allocated block ID 5298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5298 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5298 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5298.1709794002.7569551.submit parsl.executors.status_handling INFO: Allocated block ID 5299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5299 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5299 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5299.1709794002.757463.submit parsl.executors.status_handling INFO: Allocated block ID 5300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5300 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5300 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5300.1709794002.7581272.submit parsl.executors.status_handling INFO: Allocated block ID 5301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5301 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5301 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5301.1709794002.758707.submit parsl.executors.status_handling INFO: Allocated block ID 5302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5302 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5302 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5302.1709794002.7594886.submit parsl.executors.status_handling INFO: Allocated block ID 5303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5303 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5303 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5303.1709794002.7600632.submit parsl.executors.status_handling INFO: Allocated block ID 5304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5304 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5304 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5304.1709794002.7606294.submit parsl.executors.status_handling INFO: Allocated block ID 5305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5305 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5305 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5305.1709794002.7611883.submit parsl.executors.status_handling INFO: Allocated block ID 5306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5306 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5306 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5306.1709794002.7617466.submit parsl.executors.status_handling INFO: Allocated block ID 5307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5307 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5307 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5307.1709794002.7623324.submit parsl.executors.status_handling INFO: Allocated block ID 5308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5308 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5308 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5308.1709794002.7628856.submit parsl.executors.status_handling INFO: Allocated block ID 5309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5309 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5309 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5309.1709794002.763431.submit parsl.executors.status_handling INFO: Allocated block ID 5310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5310 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5310 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5310.1709794002.7639873.submit parsl.executors.status_handling INFO: Allocated block ID 5311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5311 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5311 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5311.1709794002.7645333.submit parsl.executors.status_handling INFO: Allocated block ID 5312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5312 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5312 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5312.1709794002.7650898.submit parsl.executors.status_handling INFO: Allocated block ID 5313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5313 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5313 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5313.1709794002.7656424.submit parsl.executors.status_handling INFO: Allocated block ID 5314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5314 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5314 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5314.1709794002.766357.submit parsl.executors.status_handling INFO: Allocated block ID 5315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5315 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5315 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5315.1709794002.7669277.submit parsl.executors.status_handling INFO: Allocated block ID 5316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5316 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5316 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5316.1709794002.7674823.submit parsl.executors.status_handling INFO: Allocated block ID 5317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5317 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5317 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5317.1709794002.7680407.submit parsl.executors.status_handling INFO: Allocated block ID 5318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5318 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5318 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5318.1709794002.768597.submit parsl.executors.status_handling INFO: Allocated block ID 5319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5319 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5319 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5319.1709794002.76916.submit parsl.executors.status_handling INFO: Allocated block ID 5320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5320 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5320 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5320.1709794002.7697139.submit parsl.executors.status_handling INFO: Allocated block ID 5321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5321 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5321 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5321.1709794002.7705584.submit parsl.executors.status_handling INFO: Allocated block ID 5322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5322 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5322 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5322.1709794002.77113.submit parsl.executors.status_handling INFO: Allocated block ID 5323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5323 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5323 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5323.1709794002.7716494.submit parsl.executors.status_handling INFO: Allocated block ID 5324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5324 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5324 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5324.1709794002.7721786.submit parsl.executors.status_handling INFO: Allocated block ID 5325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5325 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5325 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5325.1709794002.772697.submit parsl.executors.status_handling INFO: Allocated block ID 5326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5326 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5326 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5326.1709794002.7732594.submit parsl.executors.status_handling INFO: Allocated block ID 5327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5327 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5327 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5327.1709794002.773811.submit parsl.executors.status_handling INFO: Allocated block ID 5328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5328 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5328 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5328.1709794002.7743504.submit parsl.executors.status_handling INFO: Allocated block ID 5329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5329 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5329 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5329.1709794002.7748811.submit parsl.executors.status_handling INFO: Allocated block ID 5330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5330 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5330 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5330.1709794002.7754314.submit parsl.executors.status_handling INFO: Allocated block ID 5331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5331 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5331 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5331.1709794002.7759845.submit parsl.executors.status_handling INFO: Allocated block ID 5332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5332 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5332 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5332.1709794002.776512.submit parsl.executors.status_handling INFO: Allocated block ID 5333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5333 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5333 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5333.1709794002.7770267.submit parsl.executors.status_handling INFO: Allocated block ID 5334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5334 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5334 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5334.1709794002.7775369.submit parsl.executors.status_handling INFO: Allocated block ID 5335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5335 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5335 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5335.1709794002.7782373.submit parsl.executors.status_handling INFO: Allocated block ID 5336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5336 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5336 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5336.1709794002.7787776.submit parsl.executors.status_handling INFO: Allocated block ID 5337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5337 2024-03-06 22:46:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5337 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5337.1709794002.7794902.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d205e750a34f2564bfa595843cc7f9b with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fffe3773-5214-456e-882b-327980d225bc_biasIsr_1123837_8.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123837/fffe3773-5214-456e-882b-327980d225bc_biasIsr_1123837_8.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f2ed931af3729e13b224231fdf3c5bd with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/54522816-2b74-4af4-b7fa-7b5a33cde0c5_biasIsr_1123737_8.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123737/54522816-2b74-4af4-b7fa-7b5a33cde0c5_biasIsr_1123737_8.stderr 2024-03-06 22:46:47 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1337 parsl.dataflow.strategy DEBUG: Executor multi has 1337 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 5338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5338 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5338 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5338.1709794007.8545263.submit parsl.executors.status_handling INFO: Allocated block ID 5339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5339 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5339 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5339.1709794007.8550673.submit parsl.executors.status_handling INFO: Allocated block ID 5340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5340 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5340 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5340.1709794007.8555164.submit parsl.executors.status_handling INFO: Allocated block ID 5341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5341 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5341 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5341.1709794007.8559468.submit parsl.executors.status_handling INFO: Allocated block ID 5342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5342 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5342 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5342.1709794007.8563712.submit parsl.executors.status_handling INFO: Allocated block ID 5343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5343 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5343 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5343.1709794007.856794.submit parsl.executors.status_handling INFO: Allocated block ID 5344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5344 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5344 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5344.1709794007.8572311.submit parsl.executors.status_handling INFO: Allocated block ID 5345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5345 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5345 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5345.1709794007.857652.submit parsl.executors.status_handling INFO: Allocated block ID 5346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5346 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5346 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5346.1709794007.8580818.submit parsl.executors.status_handling INFO: Allocated block ID 5347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5347 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5347 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5347.1709794007.8585076.submit parsl.executors.status_handling INFO: Allocated block ID 5348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5348 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5348 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5348.1709794007.8589416.submit parsl.executors.status_handling INFO: Allocated block ID 5349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5349 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5349 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5349.1709794007.8593712.submit parsl.executors.status_handling INFO: Allocated block ID 5350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5350 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5350 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5350.1709794007.8597975.submit parsl.executors.status_handling INFO: Allocated block ID 5351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5351 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5351 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5351.1709794007.8602395.submit parsl.executors.status_handling INFO: Allocated block ID 5352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5352 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5352 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5352.1709794007.8606694.submit parsl.executors.status_handling INFO: Allocated block ID 5353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5353 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5353 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5353.1709794007.8610952.submit parsl.executors.status_handling INFO: Allocated block ID 5354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5354 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5354 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5354.1709794007.8615177.submit parsl.executors.status_handling INFO: Allocated block ID 5355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5355 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5355 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5355.1709794007.8619485.submit parsl.executors.status_handling INFO: Allocated block ID 5356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5356 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5356 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5356.1709794007.8623762.submit parsl.executors.status_handling INFO: Allocated block ID 5357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5357 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5357 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5357.1709794007.8627896.submit parsl.executors.status_handling INFO: Allocated block ID 5358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5358 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5358 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5358.1709794007.8632128.submit parsl.executors.status_handling INFO: Allocated block ID 5359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5359 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5359 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5359.1709794007.8636272.submit parsl.executors.status_handling INFO: Allocated block ID 5360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5360 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5360 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5360.1709794007.864053.submit parsl.executors.status_handling INFO: Allocated block ID 5361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5361 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5361 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5361.1709794007.864473.submit parsl.executors.status_handling INFO: Allocated block ID 5362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5362 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5362 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5362.1709794007.864898.submit parsl.executors.status_handling INFO: Allocated block ID 5363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5363 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5363 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5363.1709794007.8653235.submit parsl.executors.status_handling INFO: Allocated block ID 5364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5364 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5364 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5364.1709794007.8657439.submit parsl.executors.status_handling INFO: Allocated block ID 5365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5365 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5365 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5365.1709794007.8661788.submit parsl.executors.status_handling INFO: Allocated block ID 5366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5366 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5366 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5366.1709794007.866593.submit parsl.executors.status_handling INFO: Allocated block ID 5367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5367 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5367 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5367.1709794007.867017.submit parsl.executors.status_handling INFO: Allocated block ID 5368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5368 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5368 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5368.1709794007.8674383.submit parsl.executors.status_handling INFO: Allocated block ID 5369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5369 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5369 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5369.1709794007.8678584.submit parsl.executors.status_handling INFO: Allocated block ID 5370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5370 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5370 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5370.1709794007.868277.submit parsl.executors.status_handling INFO: Allocated block ID 5371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5371 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5371 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5371.1709794007.8686945.submit parsl.executors.status_handling INFO: Allocated block ID 5372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5372 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5372 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5372.1709794007.8691182.submit parsl.executors.status_handling INFO: Allocated block ID 5373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5373 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5373 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5373.1709794007.8695343.submit parsl.executors.status_handling INFO: Allocated block ID 5374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5374 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5374 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5374.1709794007.8699648.submit parsl.executors.status_handling INFO: Allocated block ID 5375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5375 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5375 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5375.1709794007.8703885.submit parsl.executors.status_handling INFO: Allocated block ID 5376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5376 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5376 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5376.1709794007.8708134.submit parsl.executors.status_handling INFO: Allocated block ID 5377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5377 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5377 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5377.1709794007.871238.submit parsl.executors.status_handling INFO: Allocated block ID 5378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5378 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5378 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5378.1709794007.8716588.submit parsl.executors.status_handling INFO: Allocated block ID 5379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5379 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5379 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5379.1709794007.8720846.submit parsl.executors.status_handling INFO: Allocated block ID 5380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5380 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5380 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5380.1709794007.8725066.submit parsl.executors.status_handling INFO: Allocated block ID 5381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5381 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5381 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5381.1709794007.872931.submit parsl.executors.status_handling INFO: Allocated block ID 5382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5382 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5382 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5382.1709794007.8733528.submit parsl.executors.status_handling INFO: Allocated block ID 5383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5383 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5383 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5383.1709794007.8737752.submit parsl.executors.status_handling INFO: Allocated block ID 5384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5384 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5384 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5384.1709794007.8742087.submit parsl.executors.status_handling INFO: Allocated block ID 5385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5385 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5385 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5385.1709794007.874629.submit parsl.executors.status_handling INFO: Allocated block ID 5386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5386 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5386 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5386.1709794007.875061.submit parsl.executors.status_handling INFO: Allocated block ID 5387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5387 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5387 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5387.1709794007.8754907.submit parsl.executors.status_handling INFO: Allocated block ID 5388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5388 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5388 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5388.1709794007.8759186.submit parsl.executors.status_handling INFO: Allocated block ID 5389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5389 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5389 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5389.1709794007.8763425.submit parsl.executors.status_handling INFO: Allocated block ID 5390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5390 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5390 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5390.1709794007.8767586.submit parsl.executors.status_handling INFO: Allocated block ID 5391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5391 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5391 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5391.1709794007.8771913.submit parsl.executors.status_handling INFO: Allocated block ID 5392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5392 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5392 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5392.1709794007.877609.submit parsl.executors.status_handling INFO: Allocated block ID 5393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5393 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5393 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5393.1709794007.8780327.submit parsl.executors.status_handling INFO: Allocated block ID 5394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5394 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5394 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5394.1709794007.8784466.submit parsl.executors.status_handling INFO: Allocated block ID 5395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5395 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5395 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5395.1709794007.8788671.submit parsl.executors.status_handling INFO: Allocated block ID 5396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5396 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5396 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5396.1709794007.8792856.submit parsl.executors.status_handling INFO: Allocated block ID 5397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5397 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5397 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5397.1709794007.8797054.submit parsl.executors.status_handling INFO: Allocated block ID 5398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5398 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5398 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5398.1709794007.8801312.submit parsl.executors.status_handling INFO: Allocated block ID 5399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5399 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5399 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5399.1709794007.8805528.submit parsl.executors.status_handling INFO: Allocated block ID 5400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5400 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5400 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5400.1709794007.8809767.submit parsl.executors.status_handling INFO: Allocated block ID 5401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5401 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5401 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5401.1709794007.8813927.submit parsl.executors.status_handling INFO: Allocated block ID 5402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5402 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5402 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5402.1709794007.8818111.submit parsl.executors.status_handling INFO: Allocated block ID 5403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5403 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5403 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5403.1709794007.882226.submit parsl.executors.status_handling INFO: Allocated block ID 5404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5404 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5404 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5404.1709794007.8826427.submit parsl.executors.status_handling INFO: Allocated block ID 5405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5405 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5405 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5405.1709794007.883061.submit parsl.executors.status_handling INFO: Allocated block ID 5406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5406 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5406 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5406.1709794007.8834767.submit parsl.executors.status_handling INFO: Allocated block ID 5407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5407 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5407 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5407.1709794007.8838959.submit parsl.executors.status_handling INFO: Allocated block ID 5408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5408 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5408 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5408.1709794007.8843093.submit parsl.executors.status_handling INFO: Allocated block ID 5409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5409 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5409 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5409.1709794007.8847263.submit parsl.executors.status_handling INFO: Allocated block ID 5410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5410 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5410 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5410.1709794007.8851569.submit parsl.executors.status_handling INFO: Allocated block ID 5411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5411 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5411 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5411.1709794007.8855793.submit parsl.executors.status_handling INFO: Allocated block ID 5412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5412 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5412 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5412.1709794007.8860157.submit parsl.executors.status_handling INFO: Allocated block ID 5413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5413 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5413 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5413.1709794007.8864377.submit parsl.executors.status_handling INFO: Allocated block ID 5414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5414 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5414 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5414.1709794007.88686.submit parsl.executors.status_handling INFO: Allocated block ID 5415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5415 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5415 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5415.1709794007.8872836.submit parsl.executors.status_handling INFO: Allocated block ID 5416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5416 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5416 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5416.1709794007.8877087.submit parsl.executors.status_handling INFO: Allocated block ID 5417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5417 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5417 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5417.1709794007.8881364.submit parsl.executors.status_handling INFO: Allocated block ID 5418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5418 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5418 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5418.1709794007.8885553.submit parsl.executors.status_handling INFO: Allocated block ID 5419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5419 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5419 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5419.1709794007.8889787.submit parsl.executors.status_handling INFO: Allocated block ID 5420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5420 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5420 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5420.1709794007.889399.submit parsl.executors.status_handling INFO: Allocated block ID 5421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5421 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5421 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5421.1709794007.8898275.submit parsl.executors.status_handling INFO: Allocated block ID 5422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5422 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5422 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5422.1709794007.8902547.submit parsl.executors.status_handling INFO: Allocated block ID 5423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5423 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5423 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5423.1709794007.8906806.submit parsl.executors.status_handling INFO: Allocated block ID 5424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5424 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5424 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5424.1709794007.8911028.submit parsl.executors.status_handling INFO: Allocated block ID 5425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5425 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5425 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5425.1709794007.8915253.submit parsl.executors.status_handling INFO: Allocated block ID 5426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5426 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5426 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5426.1709794007.891945.submit parsl.executors.status_handling INFO: Allocated block ID 5427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5427 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5427 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5427.1709794007.892364.submit parsl.executors.status_handling INFO: Allocated block ID 5428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5428 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5428 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5428.1709794007.8927844.submit parsl.executors.status_handling INFO: Allocated block ID 5429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5429 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5429 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5429.1709794007.8932085.submit parsl.executors.status_handling INFO: Allocated block ID 5430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5430 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5430 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5430.1709794007.893628.submit parsl.executors.status_handling INFO: Allocated block ID 5431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5431 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5431 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5431.1709794007.8940606.submit parsl.executors.status_handling INFO: Allocated block ID 5432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5432 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5432 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5432.1709794007.8944893.submit parsl.executors.status_handling INFO: Allocated block ID 5433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5433 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5433 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5433.1709794007.8949192.submit parsl.executors.status_handling INFO: Allocated block ID 5434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5434 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5434 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5434.1709794007.8953452.submit parsl.executors.status_handling INFO: Allocated block ID 5435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5435 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5435 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5435.1709794007.8957777.submit parsl.executors.status_handling INFO: Allocated block ID 5436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5436 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5436 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5436.1709794007.8962076.submit parsl.executors.status_handling INFO: Allocated block ID 5437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5437 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5437 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5437.1709794007.8966262.submit parsl.executors.status_handling INFO: Allocated block ID 5438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5438 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5438 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5438.1709794007.8970532.submit parsl.executors.status_handling INFO: Allocated block ID 5439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5439 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5439 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5439.1709794007.8974738.submit parsl.executors.status_handling INFO: Allocated block ID 5440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5440 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5440 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5440.1709794007.8979235.submit parsl.executors.status_handling INFO: Allocated block ID 5441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5441 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5441 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5441.1709794007.8983588.submit parsl.executors.status_handling INFO: Allocated block ID 5442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5442 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5442 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5442.1709794007.8987918.submit parsl.executors.status_handling INFO: Allocated block ID 5443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5443 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5443 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5443.1709794007.8992302.submit parsl.executors.status_handling INFO: Allocated block ID 5444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5444 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5444 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5444.1709794007.8996513.submit parsl.executors.status_handling INFO: Allocated block ID 5445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5445 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5445 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5445.1709794007.9000695.submit parsl.executors.status_handling INFO: Allocated block ID 5446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5446 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5446 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5446.1709794007.9004862.submit parsl.executors.status_handling INFO: Allocated block ID 5447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5447 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5447 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5447.1709794007.9009132.submit parsl.executors.status_handling INFO: Allocated block ID 5448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5448 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5448 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5448.1709794007.9013357.submit parsl.executors.status_handling INFO: Allocated block ID 5449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5449 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5449 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5449.1709794007.901758.submit parsl.executors.status_handling INFO: Allocated block ID 5450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5450 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5450 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5450.1709794007.902187.submit parsl.executors.status_handling INFO: Allocated block ID 5451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5451 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5451 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5451.1709794007.9026024.submit parsl.executors.status_handling INFO: Allocated block ID 5452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5452 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5452 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5452.1709794007.9030256.submit parsl.executors.status_handling INFO: Allocated block ID 5453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5453 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5453 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5453.1709794007.90346.submit parsl.executors.status_handling INFO: Allocated block ID 5454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5454 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5454 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5454.1709794007.903888.submit parsl.executors.status_handling INFO: Allocated block ID 5455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5455 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5455 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5455.1709794007.9043143.submit parsl.executors.status_handling INFO: Allocated block ID 5456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5456 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5456 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5456.1709794007.904737.submit parsl.executors.status_handling INFO: Allocated block ID 5457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5457 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5457 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5457.1709794007.9051712.submit parsl.executors.status_handling INFO: Allocated block ID 5458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5458 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5458 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5458.1709794007.9056022.submit parsl.executors.status_handling INFO: Allocated block ID 5459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5459 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5459 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5459.1709794007.9060433.submit parsl.executors.status_handling INFO: Allocated block ID 5460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5460 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5460 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5460.1709794007.9064739.submit parsl.executors.status_handling INFO: Allocated block ID 5461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5461 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5461 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5461.1709794007.9069088.submit parsl.executors.status_handling INFO: Allocated block ID 5462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5462 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5462 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5462.1709794007.9073374.submit parsl.executors.status_handling INFO: Allocated block ID 5463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5463 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5463 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5463.1709794007.908053.submit parsl.executors.status_handling INFO: Allocated block ID 5464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5464 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5464 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5464.1709794007.9084997.submit parsl.executors.status_handling INFO: Allocated block ID 5465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5465 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5465 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5465.1709794007.9089346.submit parsl.executors.status_handling INFO: Allocated block ID 5466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5466 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5466 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5466.1709794007.9093676.submit parsl.executors.status_handling INFO: Allocated block ID 5467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5467 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5467 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5467.1709794007.9097838.submit parsl.executors.status_handling INFO: Allocated block ID 5468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5468 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5468 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5468.1709794007.9102125.submit parsl.executors.status_handling INFO: Allocated block ID 5469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5469 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5469 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5469.1709794007.9106312.submit parsl.executors.status_handling INFO: Allocated block ID 5470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5470 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5470 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5470.1709794007.9110608.submit parsl.executors.status_handling INFO: Allocated block ID 5471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5471 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5471 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5471.1709794007.911485.submit parsl.executors.status_handling INFO: Allocated block ID 5472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5472 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5472 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5472.1709794007.9119213.submit parsl.executors.status_handling INFO: Allocated block ID 5473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5473 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5473 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5473.1709794007.9123464.submit parsl.executors.status_handling INFO: Allocated block ID 5474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5474 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5474 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5474.1709794007.9127662.submit parsl.executors.status_handling INFO: Allocated block ID 5475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5475 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5475 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5475.1709794007.9131892.submit parsl.executors.status_handling INFO: Allocated block ID 5476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5476 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5476 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5476.1709794007.9136112.submit parsl.executors.status_handling INFO: Allocated block ID 5477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5477 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5477 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5477.1709794007.9140399.submit parsl.executors.status_handling INFO: Allocated block ID 5478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5478 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5478 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5478.1709794007.9144616.submit parsl.executors.status_handling INFO: Allocated block ID 5479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5479 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5479 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5479.1709794007.9148877.submit parsl.executors.status_handling INFO: Allocated block ID 5480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5480 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5480 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5480.1709794007.9153094.submit parsl.executors.status_handling INFO: Allocated block ID 5481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5481 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5481 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5481.1709794007.9157317.submit parsl.executors.status_handling INFO: Allocated block ID 5482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5482 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5482 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5482.1709794007.916167.submit parsl.executors.status_handling INFO: Allocated block ID 5483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5483 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5483 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5483.1709794007.9165854.submit parsl.executors.status_handling INFO: Allocated block ID 5484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5484 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5484 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5484.1709794007.9170156.submit parsl.executors.status_handling INFO: Allocated block ID 5485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5485 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5485 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5485.1709794007.9174342.submit parsl.executors.status_handling INFO: Allocated block ID 5486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5486 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5486 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5486.1709794007.9178638.submit parsl.executors.status_handling INFO: Allocated block ID 5487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5487 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5487 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5487.1709794007.9182885.submit parsl.executors.status_handling INFO: Allocated block ID 5488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5488 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5488 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5488.1709794007.9189157.submit parsl.executors.status_handling INFO: Allocated block ID 5489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5489 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5489 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5489.1709794007.9193447.submit parsl.executors.status_handling INFO: Allocated block ID 5490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5490 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5490 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5490.1709794007.919765.submit parsl.executors.status_handling INFO: Allocated block ID 5491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5491 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5491 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5491.1709794007.9201918.submit parsl.executors.status_handling INFO: Allocated block ID 5492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5492 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5492 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5492.1709794007.9206057.submit parsl.executors.status_handling INFO: Allocated block ID 5493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5493 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5493 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5493.1709794007.921025.submit parsl.executors.status_handling INFO: Allocated block ID 5494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5494 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5494 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5494.1709794007.9214408.submit parsl.executors.status_handling INFO: Allocated block ID 5495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5495 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5495 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5495.1709794007.9218712.submit parsl.executors.status_handling INFO: Allocated block ID 5496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5496 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5496 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5496.1709794007.9222896.submit parsl.executors.status_handling INFO: Allocated block ID 5497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5497 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5497 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5497.1709794007.922703.submit parsl.executors.status_handling INFO: Allocated block ID 5498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5498 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5498 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5498.1709794007.9231226.submit parsl.executors.status_handling INFO: Allocated block ID 5499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5499 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5499 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5499.1709794007.9235327.submit parsl.executors.status_handling INFO: Allocated block ID 5500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5500 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5500 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5500.1709794007.9239585.submit parsl.executors.status_handling INFO: Allocated block ID 5501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5501 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5501 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5501.1709794007.9243774.submit parsl.executors.status_handling INFO: Allocated block ID 5502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5502 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5502 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5502.1709794007.9247925.submit parsl.executors.status_handling INFO: Allocated block ID 5503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5503 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5503 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5503.1709794007.9252408.submit parsl.executors.status_handling INFO: Allocated block ID 5504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5504 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5504 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5504.1709794007.9256644.submit parsl.executors.status_handling INFO: Allocated block ID 5505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5505 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5505 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5505.1709794007.9261062.submit parsl.executors.status_handling INFO: Allocated block ID 5506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5506 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5506 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5506.1709794007.9265263.submit parsl.executors.status_handling INFO: Allocated block ID 5507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5507 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5507 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5507.1709794007.9269538.submit parsl.executors.status_handling INFO: Allocated block ID 5508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5508 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5508 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5508.1709794007.9273717.submit parsl.executors.status_handling INFO: Allocated block ID 5509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5509 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5509 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5509.1709794007.9277842.submit parsl.executors.status_handling INFO: Allocated block ID 5510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5510 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5510 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5510.1709794007.9282045.submit parsl.executors.status_handling INFO: Allocated block ID 5511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5511 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5511 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5511.1709794007.928621.submit parsl.executors.status_handling INFO: Allocated block ID 5512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5512 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5512 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5512.1709794007.9290454.submit parsl.executors.status_handling INFO: Allocated block ID 5513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5513 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5513 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5513.1709794007.929459.submit parsl.executors.status_handling INFO: Allocated block ID 5514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5514 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5514 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5514.1709794007.9298813.submit parsl.executors.status_handling INFO: Allocated block ID 5515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5515 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5515 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5515.1709794007.930299.submit parsl.executors.status_handling INFO: Allocated block ID 5516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5516 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5516 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5516.1709794007.9307172.submit parsl.executors.status_handling INFO: Allocated block ID 5517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5517 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5517 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5517.1709794007.9311469.submit parsl.executors.status_handling INFO: Allocated block ID 5518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5518 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5518 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5518.1709794007.9315734.submit parsl.executors.status_handling INFO: Allocated block ID 5519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5519 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5519 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5519.1709794007.9320042.submit parsl.executors.status_handling INFO: Allocated block ID 5520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5520 2024-03-06 22:46:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5520 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5520.1709794007.9324267.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1337 parsl.dataflow.strategy DEBUG: Executor multi has 1337 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 5521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5521 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5521 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5521.1709794012.7400653.submit parsl.executors.status_handling INFO: Allocated block ID 5522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5522 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5522 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5522.1709794012.7405615.submit parsl.executors.status_handling INFO: Allocated block ID 5523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5523 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5523 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5523.1709794012.7410007.submit parsl.executors.status_handling INFO: Allocated block ID 5524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5524 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5524 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5524.1709794012.7414248.submit parsl.executors.status_handling INFO: Allocated block ID 5525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5525 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5525 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5525.1709794012.741855.submit parsl.executors.status_handling INFO: Allocated block ID 5526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5526 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5526 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5526.1709794012.742279.submit parsl.executors.status_handling INFO: Allocated block ID 5527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5527 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5527 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5527.1709794012.7426946.submit parsl.executors.status_handling INFO: Allocated block ID 5528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5528 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5528 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5528.1709794012.7431254.submit parsl.executors.status_handling INFO: Allocated block ID 5529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5529 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5529 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5529.1709794012.74355.submit parsl.executors.status_handling INFO: Allocated block ID 5530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5530 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5530 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5530.1709794012.7439778.submit parsl.executors.status_handling INFO: Allocated block ID 5531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5531 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5531 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5531.1709794012.7443993.submit parsl.executors.status_handling INFO: Allocated block ID 5532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5532 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5532 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5532.1709794012.7448301.submit parsl.executors.status_handling INFO: Allocated block ID 5533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5533 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5533 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5533.1709794012.7452495.submit parsl.executors.status_handling INFO: Allocated block ID 5534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5534 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5534 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5534.1709794012.7456698.submit parsl.executors.status_handling INFO: Allocated block ID 5535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5535 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5535 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5535.1709794012.7460964.submit parsl.executors.status_handling INFO: Allocated block ID 5536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5536 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5536 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5536.1709794012.746513.submit parsl.executors.status_handling INFO: Allocated block ID 5537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5537 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5537 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5537.1709794012.746937.submit parsl.executors.status_handling INFO: Allocated block ID 5538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5538 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5538 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5538.1709794012.7473583.submit parsl.executors.status_handling INFO: Allocated block ID 5539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5539 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5539 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5539.1709794012.7477744.submit parsl.executors.status_handling INFO: Allocated block ID 5540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5540 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5540 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5540.1709794012.7482026.submit parsl.executors.status_handling INFO: Allocated block ID 5541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5541 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5541 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5541.1709794012.7486346.submit parsl.executors.status_handling INFO: Allocated block ID 5542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5542 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5542 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5542.1709794012.749064.submit parsl.executors.status_handling INFO: Allocated block ID 5543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5543 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5543 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5543.1709794012.7494936.submit parsl.executors.status_handling INFO: Allocated block ID 5544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5544 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5544 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5544.1709794012.7602136.submit parsl.executors.status_handling INFO: Allocated block ID 5545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5545 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5545 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5545.1709794012.7606885.submit parsl.executors.status_handling INFO: Allocated block ID 5546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5546 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5546 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5546.1709794012.7611268.submit parsl.executors.status_handling INFO: Allocated block ID 5547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5547 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5547 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5547.1709794012.7615545.submit parsl.executors.status_handling INFO: Allocated block ID 5548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5548 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5548 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5548.1709794012.761987.submit parsl.executors.status_handling INFO: Allocated block ID 5549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5549 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5549 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5549.1709794012.7624056.submit parsl.executors.status_handling INFO: Allocated block ID 5550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5550 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5550 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5550.1709794012.7628276.submit parsl.executors.status_handling INFO: Allocated block ID 5551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5551 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5551 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5551.1709794012.763247.submit parsl.executors.status_handling INFO: Allocated block ID 5552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5552 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5552 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5552.1709794012.76367.submit parsl.executors.status_handling INFO: Allocated block ID 5553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5553 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5553 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5553.1709794012.764101.submit parsl.executors.status_handling INFO: Allocated block ID 5554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5554 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5554 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5554.1709794012.764523.submit parsl.executors.status_handling INFO: Allocated block ID 5555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5555 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5555 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5555.1709794012.7649457.submit parsl.executors.status_handling INFO: Allocated block ID 5556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5556 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5556 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5556.1709794012.76536.submit parsl.executors.status_handling INFO: Allocated block ID 5557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5557 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5557 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5557.1709794012.765778.submit parsl.executors.status_handling INFO: Allocated block ID 5558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5558 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5558 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5558.1709794012.766213.submit parsl.executors.status_handling INFO: Allocated block ID 5559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5559 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5559 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5559.1709794012.766632.submit parsl.executors.status_handling INFO: Allocated block ID 5560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5560 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5560 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5560.1709794012.7670565.submit parsl.executors.status_handling INFO: Allocated block ID 5561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5561 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5561 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5561.1709794012.7674723.submit parsl.executors.status_handling INFO: Allocated block ID 5562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5562 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5562 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5562.1709794012.7678933.submit parsl.executors.status_handling INFO: Allocated block ID 5563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5563 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5563 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5563.1709794012.7683136.submit parsl.executors.status_handling INFO: Allocated block ID 5564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5564 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5564 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5564.1709794012.768733.submit parsl.executors.status_handling INFO: Allocated block ID 5565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5565 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5565 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5565.1709794012.7691638.submit parsl.executors.status_handling INFO: Allocated block ID 5566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5566 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5566 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5566.1709794012.7695837.submit parsl.executors.status_handling INFO: Allocated block ID 5567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5567 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5567 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5567.1709794012.7700064.submit parsl.executors.status_handling INFO: Allocated block ID 5568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5568 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5568 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5568.1709794012.7704313.submit parsl.executors.status_handling INFO: Allocated block ID 5569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5569 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5569 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5569.1709794012.7912638.submit parsl.executors.status_handling INFO: Allocated block ID 5570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5570 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5570 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5570.1709794012.791789.submit parsl.executors.status_handling INFO: Allocated block ID 5571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5571 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5571 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5571.1709794012.7922273.submit parsl.executors.status_handling INFO: Allocated block ID 5572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5572 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5572 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5572.1709794012.7926464.submit parsl.executors.status_handling INFO: Allocated block ID 5573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5573 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5573 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5573.1709794012.793069.submit parsl.executors.status_handling INFO: Allocated block ID 5574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5574 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5574 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5574.1709794012.7934895.submit parsl.executors.status_handling INFO: Allocated block ID 5575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5575 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5575 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5575.1709794012.7939172.submit parsl.executors.status_handling INFO: Allocated block ID 5576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5576 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5576 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5576.1709794012.7943385.submit parsl.executors.status_handling INFO: Allocated block ID 5577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5577 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5577 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5577.1709794012.7947714.submit parsl.executors.status_handling INFO: Allocated block ID 5578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5578 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5578 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5578.1709794012.7952027.submit parsl.executors.status_handling INFO: Allocated block ID 5579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5579 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5579 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5579.1709794012.795622.submit parsl.executors.status_handling INFO: Allocated block ID 5580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5580 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5580 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5580.1709794012.796049.submit parsl.executors.status_handling INFO: Allocated block ID 5581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5581 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5581 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5581.1709794012.7964692.submit parsl.executors.status_handling INFO: Allocated block ID 5582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5582 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5582 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5582.1709794012.796898.submit parsl.executors.status_handling INFO: Allocated block ID 5583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5583 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5583 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5583.1709794012.797316.submit parsl.executors.status_handling INFO: Allocated block ID 5584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5584 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5584 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5584.1709794012.7977326.submit parsl.executors.status_handling INFO: Allocated block ID 5585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5585 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5585 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5585.1709794012.7981572.submit parsl.executors.status_handling INFO: Allocated block ID 5586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5586 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5586 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5586.1709794012.798576.submit parsl.executors.status_handling INFO: Allocated block ID 5587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5587 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5587 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5587.1709794012.7989955.submit parsl.executors.status_handling INFO: Allocated block ID 5588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5588 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5588 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5588.1709794012.7994134.submit parsl.executors.status_handling INFO: Allocated block ID 5589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5589 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5589 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5589.1709794012.79984.submit parsl.executors.status_handling INFO: Allocated block ID 5590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5590 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5590 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5590.1709794012.8002598.submit parsl.executors.status_handling INFO: Allocated block ID 5591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5591 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5591 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5591.1709794012.8006778.submit parsl.executors.status_handling INFO: Allocated block ID 5592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5592 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5592 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5592.1709794012.8011117.submit parsl.executors.status_handling INFO: Allocated block ID 5593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5593 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5593 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5593.1709794012.8015368.submit parsl.executors.status_handling INFO: Allocated block ID 5594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5594 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5594 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5594.1709794012.8122694.submit parsl.executors.status_handling INFO: Allocated block ID 5595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5595 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5595 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5595.1709794012.8127394.submit parsl.executors.status_handling INFO: Allocated block ID 5596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5596 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5596 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5596.1709794012.8131788.submit parsl.executors.status_handling INFO: Allocated block ID 5597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5597 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5597 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5597.1709794012.813597.submit parsl.executors.status_handling INFO: Allocated block ID 5598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5598 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5598 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5598.1709794012.8140185.submit parsl.executors.status_handling INFO: Allocated block ID 5599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5599 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5599 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5599.1709794012.8144352.submit parsl.executors.status_handling INFO: Allocated block ID 5600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5600 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5600 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5600.1709794012.8148613.submit parsl.executors.status_handling INFO: Allocated block ID 5601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5601 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5601 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5601.1709794012.815293.submit parsl.executors.status_handling INFO: Allocated block ID 5602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5602 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5602 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5602.1709794012.8157175.submit parsl.executors.status_handling INFO: Allocated block ID 5603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5603 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5603 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5603.1709794012.8161483.submit parsl.executors.status_handling INFO: Allocated block ID 5604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5604 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5604 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5604.1709794012.8165693.submit parsl.executors.status_handling INFO: Allocated block ID 5605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5605 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5605 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5605.1709794012.8169954.submit parsl.executors.status_handling INFO: Allocated block ID 5606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5606 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5606 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5606.1709794012.8174164.submit parsl.executors.status_handling INFO: Allocated block ID 5607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5607 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5607 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5607.1709794012.817842.submit parsl.executors.status_handling INFO: Allocated block ID 5608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5608 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5608 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5608.1709794012.8182623.submit parsl.executors.status_handling INFO: Allocated block ID 5609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5609 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5609 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5609.1709794012.8186789.submit parsl.executors.status_handling INFO: Allocated block ID 5610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5610 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5610 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5610.1709794012.8191006.submit parsl.executors.status_handling INFO: Allocated block ID 5611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5611 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5611 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5611.1709794012.8195138.submit parsl.executors.status_handling INFO: Allocated block ID 5612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5612 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5612 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5612.1709794012.819939.submit parsl.executors.status_handling INFO: Allocated block ID 5613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5613 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5613 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5613.1709794012.8203588.submit parsl.executors.status_handling INFO: Allocated block ID 5614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5614 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5614 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5614.1709794012.8207748.submit parsl.executors.status_handling INFO: Allocated block ID 5615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5615 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5615 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5615.1709794012.8211987.submit parsl.executors.status_handling INFO: Allocated block ID 5616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5616 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5616 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5616.1709794012.8216145.submit parsl.executors.status_handling INFO: Allocated block ID 5617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5617 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5617 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5617.1709794012.8220348.submit parsl.executors.status_handling INFO: Allocated block ID 5618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5618 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5618 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5618.1709794012.8224616.submit parsl.executors.status_handling INFO: Allocated block ID 5619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5619 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5619 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5619.1709794012.8332016.submit parsl.executors.status_handling INFO: Allocated block ID 5620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5620 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5620 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5620.1709794012.8336954.submit parsl.executors.status_handling INFO: Allocated block ID 5621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5621 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5621 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5621.1709794012.8341296.submit parsl.executors.status_handling INFO: Allocated block ID 5622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5622 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5622 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5622.1709794012.8345566.submit parsl.executors.status_handling INFO: Allocated block ID 5623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5623 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5623 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5623.1709794012.834981.submit parsl.executors.status_handling INFO: Allocated block ID 5624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5624 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5624 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5624.1709794012.835403.submit parsl.executors.status_handling INFO: Allocated block ID 5625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5625 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5625 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5625.1709794012.8358297.submit parsl.executors.status_handling INFO: Allocated block ID 5626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5626 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5626 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5626.1709794012.836254.submit parsl.executors.status_handling INFO: Allocated block ID 5627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5627 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5627 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5627.1709794012.8366659.submit parsl.executors.status_handling INFO: Allocated block ID 5628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5628 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5628 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5628.1709794012.837104.submit parsl.executors.status_handling INFO: Allocated block ID 5629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5629 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5629 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5629.1709794012.837529.submit parsl.executors.status_handling INFO: Allocated block ID 5630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5630 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5630 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5630.1709794012.8379629.submit parsl.executors.status_handling INFO: Allocated block ID 5631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5631 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5631 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5631.1709794012.8383977.submit parsl.executors.status_handling INFO: Allocated block ID 5632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5632 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5632 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5632.1709794012.8388183.submit parsl.executors.status_handling INFO: Allocated block ID 5633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5633 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5633 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5633.1709794012.8392324.submit parsl.executors.status_handling INFO: Allocated block ID 5634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5634 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5634 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5634.1709794012.8396475.submit parsl.executors.status_handling INFO: Allocated block ID 5635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5635 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5635 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5635.1709794012.8400652.submit parsl.executors.status_handling INFO: Allocated block ID 5636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5636 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5636 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5636.1709794012.8404872.submit parsl.executors.status_handling INFO: Allocated block ID 5637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5637 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5637 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5637.1709794012.8409235.submit parsl.executors.status_handling INFO: Allocated block ID 5638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5638 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5638 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5638.1709794012.8413453.submit parsl.executors.status_handling INFO: Allocated block ID 5639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5639 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5639 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5639.1709794012.841763.submit parsl.executors.status_handling INFO: Allocated block ID 5640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5640 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5640 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5640.1709794012.8421857.submit parsl.executors.status_handling INFO: Allocated block ID 5641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5641 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5641 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5641.1709794012.842603.submit parsl.executors.status_handling INFO: Allocated block ID 5642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5642 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5642 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5642.1709794012.8430455.submit parsl.executors.status_handling INFO: Allocated block ID 5643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5643 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5643 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5643.1709794012.8434823.submit parsl.executors.status_handling INFO: Allocated block ID 5644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5644 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5644 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5644.1709794012.854244.submit parsl.executors.status_handling INFO: Allocated block ID 5645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5645 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5645 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5645.1709794012.8547187.submit parsl.executors.status_handling INFO: Allocated block ID 5646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5646 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5646 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5646.1709794012.8551517.submit parsl.executors.status_handling INFO: Allocated block ID 5647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5647 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5647 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5647.1709794012.855575.submit parsl.executors.status_handling INFO: Allocated block ID 5648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5648 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5648 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5648.1709794012.8560047.submit parsl.executors.status_handling INFO: Allocated block ID 5649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5649 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5649 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5649.1709794012.8564317.submit parsl.executors.status_handling INFO: Allocated block ID 5650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5650 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5650 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5650.1709794012.8568625.submit parsl.executors.status_handling INFO: Allocated block ID 5651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5651 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5651 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5651.1709794012.8572829.submit parsl.executors.status_handling INFO: Allocated block ID 5652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5652 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5652 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5652.1709794012.8577065.submit parsl.executors.status_handling INFO: Allocated block ID 5653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5653 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5653 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5653.1709794012.8581288.submit parsl.executors.status_handling INFO: Allocated block ID 5654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5654 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5654 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5654.1709794012.8585486.submit parsl.executors.status_handling INFO: Allocated block ID 5655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5655 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5655 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5655.1709794012.8589852.submit parsl.executors.status_handling INFO: Allocated block ID 5656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5656 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5656 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5656.1709794012.8594096.submit parsl.executors.status_handling INFO: Allocated block ID 5657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5657 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5657 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5657.1709794012.8598404.submit parsl.executors.status_handling INFO: Allocated block ID 5658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5658 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5658 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5658.1709794012.8602731.submit parsl.executors.status_handling INFO: Allocated block ID 5659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5659 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5659 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5659.1709794012.8606837.submit parsl.executors.status_handling INFO: Allocated block ID 5660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5660 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5660 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5660.1709794012.8611352.submit parsl.executors.status_handling INFO: Allocated block ID 5661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5661 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5661 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5661.1709794012.8615756.submit parsl.executors.status_handling INFO: Allocated block ID 5662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5662 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5662 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5662.1709794012.8620222.submit parsl.executors.status_handling INFO: Allocated block ID 5663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5663 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5663 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5663.1709794012.8624427.submit parsl.executors.status_handling INFO: Allocated block ID 5664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5664 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5664 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5664.1709794012.86286.submit parsl.executors.status_handling INFO: Allocated block ID 5665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5665 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5665 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5665.1709794012.8632846.submit parsl.executors.status_handling INFO: Allocated block ID 5666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5666 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5666 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5666.1709794012.8637016.submit parsl.executors.status_handling INFO: Allocated block ID 5667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5667 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5667 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5667.1709794012.8641334.submit parsl.executors.status_handling INFO: Allocated block ID 5668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5668 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5668 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5668.1709794012.8645494.submit parsl.executors.status_handling INFO: Allocated block ID 5669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5669 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5669 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5669.1709794012.875308.submit parsl.executors.status_handling INFO: Allocated block ID 5670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5670 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5670 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5670.1709794012.875802.submit parsl.executors.status_handling INFO: Allocated block ID 5671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5671 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5671 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5671.1709794012.8762484.submit parsl.executors.status_handling INFO: Allocated block ID 5672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5672 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5672 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5672.1709794012.8766751.submit parsl.executors.status_handling INFO: Allocated block ID 5673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5673 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5673 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5673.1709794012.87711.submit parsl.executors.status_handling INFO: Allocated block ID 5674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5674 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5674 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5674.1709794012.877531.submit parsl.executors.status_handling INFO: Allocated block ID 5675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5675 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5675 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5675.1709794012.8779578.submit parsl.executors.status_handling INFO: Allocated block ID 5676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5676 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5676 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5676.1709794012.878374.submit parsl.executors.status_handling INFO: Allocated block ID 5677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5677 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5677 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5677.1709794012.878792.submit parsl.executors.status_handling INFO: Allocated block ID 5678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5678 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5678 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5678.1709794012.8792176.submit parsl.executors.status_handling INFO: Allocated block ID 5679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5679 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5679 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5679.1709794012.8796346.submit parsl.executors.status_handling INFO: Allocated block ID 5680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5680 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5680 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5680.1709794012.8800542.submit parsl.executors.status_handling INFO: Allocated block ID 5681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5681 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5681 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5681.1709794012.8804674.submit parsl.executors.status_handling INFO: Allocated block ID 5682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5682 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5682 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5682.1709794012.8808868.submit parsl.executors.status_handling INFO: Allocated block ID 5683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5683 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5683 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5683.1709794012.881311.submit parsl.executors.status_handling INFO: Allocated block ID 5684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5684 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5684 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5684.1709794012.8817306.submit parsl.executors.status_handling INFO: Allocated block ID 5685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5685 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5685 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5685.1709794012.88216.submit parsl.executors.status_handling INFO: Allocated block ID 5686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5686 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5686 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5686.1709794012.882586.submit parsl.executors.status_handling INFO: Allocated block ID 5687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5687 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5687 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5687.1709794012.8830116.submit parsl.executors.status_handling INFO: Allocated block ID 5688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5688 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5688 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5688.1709794012.8834288.submit parsl.executors.status_handling INFO: Allocated block ID 5689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5689 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5689 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5689.1709794012.8838527.submit parsl.executors.status_handling INFO: Allocated block ID 5690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5690 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5690 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5690.1709794012.8842757.submit parsl.executors.status_handling INFO: Allocated block ID 5691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5691 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5691 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5691.1709794012.8846948.submit parsl.executors.status_handling INFO: Allocated block ID 5692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5692 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5692 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5692.1709794012.8851237.submit parsl.executors.status_handling INFO: Allocated block ID 5693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5693 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5693 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5693.1709794012.885542.submit parsl.executors.status_handling INFO: Allocated block ID 5694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5694 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5694 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5694.1709794012.913279.submit parsl.executors.status_handling INFO: Allocated block ID 5695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5695 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5695 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5695.1709794012.9137905.submit parsl.executors.status_handling INFO: Allocated block ID 5696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5696 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5696 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5696.1709794012.9142425.submit parsl.executors.status_handling INFO: Allocated block ID 5697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5697 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5697 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5697.1709794012.9146838.submit parsl.executors.status_handling INFO: Allocated block ID 5698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5698 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5698 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5698.1709794012.915115.submit parsl.executors.status_handling INFO: Allocated block ID 5699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5699 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5699 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5699.1709794012.915536.submit parsl.executors.status_handling INFO: Allocated block ID 5700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5700 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5700 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5700.1709794012.9159594.submit parsl.executors.status_handling INFO: Allocated block ID 5701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5701 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5701 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5701.1709794012.916376.submit parsl.executors.status_handling INFO: Allocated block ID 5702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5702 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5702 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5702.1709794012.9167976.submit parsl.executors.status_handling INFO: Allocated block ID 5703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5703 2024-03-06 22:46:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5703 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5703.1709794012.9172218.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19e07a2671a2fc393e8bdaa12010a573 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb312fec-cc1a-4ac1-9ad4-50a93b155d8e_biasIsr_1123839_8.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123839/fb312fec-cc1a-4ac1-9ad4-50a93b155d8e_biasIsr_1123839_8.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry affe6b447797f020279dd36c4b220d7a with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a7bfc1e9-00d6-4a68-afc0-147a04eef777_biasIsr_1123836_8.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123836/a7bfc1e9-00d6-4a68-afc0-147a04eef777_biasIsr_1123836_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1335 parsl.dataflow.strategy DEBUG: Executor multi has 1335 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 5704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5704 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5704 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5704.1709794017.7790499.submit parsl.executors.status_handling INFO: Allocated block ID 5705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5705 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5705 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5705.1709794017.7795277.submit parsl.executors.status_handling INFO: Allocated block ID 5706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5706 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5706 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5706.1709794017.779966.submit parsl.executors.status_handling INFO: Allocated block ID 5707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5707 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5707 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5707.1709794017.780397.submit parsl.executors.status_handling INFO: Allocated block ID 5708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5708 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5708 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5708.1709794017.7808228.submit parsl.executors.status_handling INFO: Allocated block ID 5709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5709 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5709 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5709.1709794017.7812455.submit parsl.executors.status_handling INFO: Allocated block ID 5710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5710 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5710 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5710.1709794017.7816713.submit parsl.executors.status_handling INFO: Allocated block ID 5711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5711 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5711 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5711.1709794017.7820966.submit parsl.executors.status_handling INFO: Allocated block ID 5712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5712 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5712 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5712.1709794017.782519.submit parsl.executors.status_handling INFO: Allocated block ID 5713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5713 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5713 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5713.1709794017.7829444.submit parsl.executors.status_handling INFO: Allocated block ID 5714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5714 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5714 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5714.1709794017.783363.submit parsl.executors.status_handling INFO: Allocated block ID 5715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5715 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5715 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5715.1709794017.7837799.submit parsl.executors.status_handling INFO: Allocated block ID 5716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5716 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5716 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5716.1709794017.7842174.submit parsl.executors.status_handling INFO: Allocated block ID 5717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5717 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5717 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5717.1709794017.7846403.submit parsl.executors.status_handling INFO: Allocated block ID 5718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5718 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5718 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5718.1709794017.785067.submit parsl.executors.status_handling INFO: Allocated block ID 5719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5719 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5719 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5719.1709794017.785485.submit parsl.executors.status_handling INFO: Allocated block ID 5720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5720 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5720 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5720.1709794017.7859144.submit parsl.executors.status_handling INFO: Allocated block ID 5721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5721 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5721 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5721.1709794017.7863371.submit parsl.executors.status_handling INFO: Allocated block ID 5722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5722 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5722 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5722.1709794017.7867565.submit parsl.executors.status_handling INFO: Allocated block ID 5723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5723 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5723 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5723.1709794017.7871795.submit parsl.executors.status_handling INFO: Allocated block ID 5724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5724 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5724 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5724.1709794017.787598.submit parsl.executors.status_handling INFO: Allocated block ID 5725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5725 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5725 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5725.1709794017.7880208.submit parsl.executors.status_handling INFO: Allocated block ID 5726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5726 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5726 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5726.1709794017.7884395.submit parsl.executors.status_handling INFO: Allocated block ID 5727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5727 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5727 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5727.1709794017.7990382.submit parsl.executors.status_handling INFO: Allocated block ID 5728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5728 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5728 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5728.1709794017.7995386.submit parsl.executors.status_handling INFO: Allocated block ID 5729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5729 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5729 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5729.1709794017.7999873.submit parsl.executors.status_handling INFO: Allocated block ID 5730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5730 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5730 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5730.1709794017.800423.submit parsl.executors.status_handling INFO: Allocated block ID 5731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5731 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5731 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5731.1709794017.8008626.submit parsl.executors.status_handling INFO: Allocated block ID 5732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5732 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5732 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5732.1709794017.8012817.submit parsl.executors.status_handling INFO: Allocated block ID 5733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5733 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5733 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5733.1709794017.8017018.submit parsl.executors.status_handling INFO: Allocated block ID 5734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5734 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5734 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5734.1709794017.8021317.submit parsl.executors.status_handling INFO: Allocated block ID 5735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5735 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5735 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5735.1709794017.802553.submit parsl.executors.status_handling INFO: Allocated block ID 5736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5736 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5736 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5736.1709794017.8029783.submit parsl.executors.status_handling INFO: Allocated block ID 5737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5737 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5737 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5737.1709794017.8033946.submit parsl.executors.status_handling INFO: Allocated block ID 5738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5738 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5738 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5738.1709794017.8038175.submit parsl.executors.status_handling INFO: Allocated block ID 5739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5739 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5739 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5739.1709794017.8042374.submit parsl.executors.status_handling INFO: Allocated block ID 5740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5740 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5740 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5740.1709794017.8046625.submit parsl.executors.status_handling INFO: Allocated block ID 5741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5741 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5741 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5741.1709794017.8050873.submit parsl.executors.status_handling INFO: Allocated block ID 5742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5742 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5742 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5742.1709794017.8055086.submit parsl.executors.status_handling INFO: Allocated block ID 5743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5743 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5743 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5743.1709794017.8059387.submit parsl.executors.status_handling INFO: Allocated block ID 5744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5744 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5744 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5744.1709794017.8063579.submit parsl.executors.status_handling INFO: Allocated block ID 5745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5745 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5745 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5745.1709794017.8067715.submit parsl.executors.status_handling INFO: Allocated block ID 5746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5746 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5746 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5746.1709794017.8071973.submit parsl.executors.status_handling INFO: Allocated block ID 5747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5747 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5747 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5747.1709794017.8076122.submit parsl.executors.status_handling INFO: Allocated block ID 5748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5748 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5748 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5748.1709794017.808035.submit parsl.executors.status_handling INFO: Allocated block ID 5749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5749 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5749 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5749.1709794017.8084562.submit parsl.executors.status_handling INFO: Allocated block ID 5750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5750 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5750 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5750.1709794017.8088894.submit parsl.executors.status_handling INFO: Allocated block ID 5751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5751 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5751 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5751.1709794017.8093195.submit parsl.executors.status_handling INFO: Allocated block ID 5752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5752 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5752 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5752.1709794017.809746.submit parsl.executors.status_handling INFO: Allocated block ID 5753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5753 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5753 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5753.1709794017.830442.submit parsl.executors.status_handling INFO: Allocated block ID 5754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5754 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5754 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5754.1709794017.8308864.submit parsl.executors.status_handling INFO: Allocated block ID 5755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5755 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5755 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5755.1709794017.831312.submit parsl.executors.status_handling INFO: Allocated block ID 5756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5756 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5756 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5756.1709794017.8317342.submit parsl.executors.status_handling INFO: Allocated block ID 5757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5757 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5757 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5757.1709794017.8321688.submit parsl.executors.status_handling INFO: Allocated block ID 5758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5758 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5758 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5758.1709794017.8325937.submit parsl.executors.status_handling INFO: Allocated block ID 5759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5759 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5759 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5759.1709794017.8330178.submit parsl.executors.status_handling INFO: Allocated block ID 5760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5760 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5760 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5760.1709794017.8334353.submit parsl.executors.status_handling INFO: Allocated block ID 5761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5761 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5761 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5761.1709794017.833855.submit parsl.executors.status_handling INFO: Allocated block ID 5762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5762 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5762 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5762.1709794017.8342755.submit parsl.executors.status_handling INFO: Allocated block ID 5763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5763 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5763 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5763.1709794017.834691.submit parsl.executors.status_handling INFO: Allocated block ID 5764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5764 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5764 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5764.1709794017.835128.submit parsl.executors.status_handling INFO: Allocated block ID 5765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5765 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5765 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5765.1709794017.8355494.submit parsl.executors.status_handling INFO: Allocated block ID 5766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5766 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5766 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5766.1709794017.8359766.submit parsl.executors.status_handling INFO: Allocated block ID 5767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5767 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5767 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5767.1709794017.8363945.submit parsl.executors.status_handling INFO: Allocated block ID 5768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5768 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5768 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5768.1709794017.8368173.submit parsl.executors.status_handling INFO: Allocated block ID 5769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5769 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5769 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5769.1709794017.837234.submit parsl.executors.status_handling INFO: Allocated block ID 5770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5770 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5770 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5770.1709794017.8376508.submit parsl.executors.status_handling INFO: Allocated block ID 5771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5771 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5771 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5771.1709794017.838084.submit parsl.executors.status_handling INFO: Allocated block ID 5772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5772 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5772 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5772.1709794017.838511.submit parsl.executors.status_handling INFO: Allocated block ID 5773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5773 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5773 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5773.1709794017.8389394.submit parsl.executors.status_handling INFO: Allocated block ID 5774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5774 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5774 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5774.1709794017.8393564.submit parsl.executors.status_handling INFO: Allocated block ID 5775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5775 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5775 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5775.1709794017.8397725.submit parsl.executors.status_handling INFO: Allocated block ID 5776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5776 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5776 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5776.1709794017.8402145.submit parsl.executors.status_handling INFO: Allocated block ID 5777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5777 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5777 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5777.1709794017.8406384.submit parsl.executors.status_handling INFO: Allocated block ID 5778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5778 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5778 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5778.1709794017.8512366.submit parsl.executors.status_handling INFO: Allocated block ID 5779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5779 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5779 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5779.1709794017.8516815.submit parsl.executors.status_handling INFO: Allocated block ID 5780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5780 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5780 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5780.1709794017.852115.submit parsl.executors.status_handling INFO: Allocated block ID 5781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5781 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5781 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5781.1709794017.852537.submit parsl.executors.status_handling INFO: Allocated block ID 5782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5782 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5782 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5782.1709794017.852986.submit parsl.executors.status_handling INFO: Allocated block ID 5783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5783 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5783 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5783.1709794017.8534153.submit parsl.executors.status_handling INFO: Allocated block ID 5784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5784 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5784 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5784.1709794017.8538516.submit parsl.executors.status_handling INFO: Allocated block ID 5785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5785 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5785 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5785.1709794017.854288.submit parsl.executors.status_handling INFO: Allocated block ID 5786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5786 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5786 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5786.1709794017.8547082.submit parsl.executors.status_handling INFO: Allocated block ID 5787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5787 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5787 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5787.1709794017.8551373.submit parsl.executors.status_handling INFO: Allocated block ID 5788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5788 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5788 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5788.1709794017.8555639.submit parsl.executors.status_handling INFO: Allocated block ID 5789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5789 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5789 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5789.1709794017.8559895.submit parsl.executors.status_handling INFO: Allocated block ID 5790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5790 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5790 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5790.1709794017.8564112.submit parsl.executors.status_handling INFO: Allocated block ID 5791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5791 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5791 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5791.1709794017.8568337.submit parsl.executors.status_handling INFO: Allocated block ID 5792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5792 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5792 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5792.1709794017.8572593.submit parsl.executors.status_handling INFO: Allocated block ID 5793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5793 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5793 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5793.1709794017.857676.submit parsl.executors.status_handling INFO: Allocated block ID 5794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5794 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5794 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5794.1709794017.8581035.submit parsl.executors.status_handling INFO: Allocated block ID 5795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5795 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5795 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5795.1709794017.8585227.submit parsl.executors.status_handling INFO: Allocated block ID 5796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5796 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5796 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5796.1709794017.8589675.submit parsl.executors.status_handling INFO: Allocated block ID 5797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5797 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5797 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5797.1709794017.8594.submit parsl.executors.status_handling INFO: Allocated block ID 5798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5798 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5798 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5798.1709794017.8598368.submit parsl.executors.status_handling INFO: Allocated block ID 5799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5799 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5799 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5799.1709794017.8602684.submit parsl.executors.status_handling INFO: Allocated block ID 5800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5800 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5800 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5800.1709794017.8607092.submit parsl.executors.status_handling INFO: Allocated block ID 5801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5801 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5801 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5801.1709794017.8611512.submit parsl.executors.status_handling INFO: Allocated block ID 5802 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5802 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5802 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5802 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5802 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5802.1709794017.8615825.submit parsl.executors.status_handling INFO: Allocated block ID 5803 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5803 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5803 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5803 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5803 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5803.1709794017.872176.submit parsl.executors.status_handling INFO: Allocated block ID 5804 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5804 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5804 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5804 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5804 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5804.1709794017.8726249.submit parsl.executors.status_handling INFO: Allocated block ID 5805 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5805 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5805 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5805 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5805 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5805.1709794017.8730567.submit parsl.executors.status_handling INFO: Allocated block ID 5806 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5806 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5806 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5806 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5806 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5806.1709794017.8734813.submit parsl.executors.status_handling INFO: Allocated block ID 5807 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5807 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5807 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5807 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5807 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5807.1709794017.8739069.submit parsl.executors.status_handling INFO: Allocated block ID 5808 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5808 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5808 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5808 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5808 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5808.1709794017.8743324.submit parsl.executors.status_handling INFO: Allocated block ID 5809 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5809 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5809 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5809 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5809 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5809.1709794017.874751.submit parsl.executors.status_handling INFO: Allocated block ID 5810 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5810 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5810 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5810 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5810 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5810.1709794017.8751793.submit parsl.executors.status_handling INFO: Allocated block ID 5811 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5811 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5811 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5811 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5811 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5811.1709794017.8755965.submit parsl.executors.status_handling INFO: Allocated block ID 5812 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5812 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5812 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5812 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5812 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5812.1709794017.8760304.submit parsl.executors.status_handling INFO: Allocated block ID 5813 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5813 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5813 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5813 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5813 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5813.1709794017.8764632.submit parsl.executors.status_handling INFO: Allocated block ID 5814 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5814 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5814 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5814 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5814 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5814.1709794017.876888.submit parsl.executors.status_handling INFO: Allocated block ID 5815 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5815 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5815 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5815 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5815 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5815.1709794017.8773034.submit parsl.executors.status_handling INFO: Allocated block ID 5816 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5816 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5816 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5816 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5816 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5816.1709794017.877717.submit parsl.executors.status_handling INFO: Allocated block ID 5817 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5817 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5817 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5817 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5817 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5817.1709794017.878145.submit parsl.executors.status_handling INFO: Allocated block ID 5818 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5818 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5818 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5818 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5818 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5818.1709794017.8785675.submit parsl.executors.status_handling INFO: Allocated block ID 5819 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5819 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5819 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5819 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5819 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5819.1709794017.878988.submit parsl.executors.status_handling INFO: Allocated block ID 5820 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5820 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5820 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5820 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5820 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5820.1709794017.8794017.submit parsl.executors.status_handling INFO: Allocated block ID 5821 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5821 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5821 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5821 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5821 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5821.1709794017.8798268.submit parsl.executors.status_handling INFO: Allocated block ID 5822 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5822 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5822 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5822 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5822 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5822.1709794017.8802438.submit parsl.executors.status_handling INFO: Allocated block ID 5823 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5823 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5823 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5823 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5823 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5823.1709794017.8806574.submit parsl.executors.status_handling INFO: Allocated block ID 5824 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5824 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5824 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5824 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5824 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5824.1709794017.8810868.submit parsl.executors.status_handling INFO: Allocated block ID 5825 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5825 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5825 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5825 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5825 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5825.1709794017.8815048.submit parsl.executors.status_handling INFO: Allocated block ID 5826 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5826 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5826 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5826 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5826 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5826.1709794017.8819273.submit parsl.executors.status_handling INFO: Allocated block ID 5827 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5827 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5827 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5827 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5827 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5827.1709794017.8823488.submit parsl.executors.status_handling INFO: Allocated block ID 5828 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5828 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5828 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5828 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5828 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5828.1709794017.882786.submit parsl.executors.status_handling INFO: Allocated block ID 5829 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5829 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5829 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5829 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5829 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5829.1709794017.8934186.submit parsl.executors.status_handling INFO: Allocated block ID 5830 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5830 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5830 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5830 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5830 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5830.1709794017.893875.submit parsl.executors.status_handling INFO: Allocated block ID 5831 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5831 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5831 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5831 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5831 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5831.1709794017.89431.submit parsl.executors.status_handling INFO: Allocated block ID 5832 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5832 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5832 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5832 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5832 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5832.1709794017.8947504.submit parsl.executors.status_handling INFO: Allocated block ID 5833 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5833 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5833 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5833 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5833 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5833.1709794017.8951886.submit parsl.executors.status_handling INFO: Allocated block ID 5834 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5834 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5834 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5834 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5834 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5834.1709794017.8956234.submit parsl.executors.status_handling INFO: Allocated block ID 5835 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5835 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5835 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5835 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5835 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5835.1709794017.896059.submit parsl.executors.status_handling INFO: Allocated block ID 5836 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5836 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5836 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5836 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5836 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5836.1709794017.8964982.submit parsl.executors.status_handling INFO: Allocated block ID 5837 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5837 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5837 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5837 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5837 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5837.1709794017.896934.submit parsl.executors.status_handling INFO: Allocated block ID 5838 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5838 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5838 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5838 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5838 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5838.1709794017.8973603.submit parsl.executors.status_handling INFO: Allocated block ID 5839 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5839 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5839 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5839 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5839 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5839.1709794017.8977754.submit parsl.executors.status_handling INFO: Allocated block ID 5840 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5840 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5840 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5840 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5840 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5840.1709794017.8982003.submit parsl.executors.status_handling INFO: Allocated block ID 5841 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5841 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5841 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5841 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5841 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5841.1709794017.898615.submit parsl.executors.status_handling INFO: Allocated block ID 5842 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5842 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5842 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5842 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5842 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5842.1709794017.899059.submit parsl.executors.status_handling INFO: Allocated block ID 5843 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5843 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5843 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5843 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5843 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5843.1709794017.8994853.submit parsl.executors.status_handling INFO: Allocated block ID 5844 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5844 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5844 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5844 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5844 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5844.1709794017.8999186.submit parsl.executors.status_handling INFO: Allocated block ID 5845 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5845 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5845 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5845 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5845 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5845.1709794017.900346.submit parsl.executors.status_handling INFO: Allocated block ID 5846 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5846 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5846 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5846 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5846 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5846.1709794017.9007602.submit parsl.executors.status_handling INFO: Allocated block ID 5847 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5847 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5847 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5847 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5847 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5847.1709794017.901186.submit parsl.executors.status_handling INFO: Allocated block ID 5848 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5848 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5848 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5848 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5848 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5848.1709794017.9016113.submit parsl.executors.status_handling INFO: Allocated block ID 5849 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5849 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5849 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5849 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5849 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5849.1709794017.9020357.submit parsl.executors.status_handling INFO: Allocated block ID 5850 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5850 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5850 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5850 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5850 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5850.1709794017.902455.submit parsl.executors.status_handling INFO: Allocated block ID 5851 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5851 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5851 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5851 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5851 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5851.1709794017.9028757.submit parsl.executors.status_handling INFO: Allocated block ID 5852 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5852 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5852 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5852 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5852 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5852.1709794017.9032955.submit parsl.executors.status_handling INFO: Allocated block ID 5853 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5853 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5853 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5853 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5853 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5853.1709794017.9037116.submit parsl.executors.status_handling INFO: Allocated block ID 5854 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5854 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5854 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5854 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5854 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5854.1709794017.9143188.submit parsl.executors.status_handling INFO: Allocated block ID 5855 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5855 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5855 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5855 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5855 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5855.1709794017.9147549.submit parsl.executors.status_handling INFO: Allocated block ID 5856 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5856 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5856 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5856 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5856 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5856.1709794017.9151938.submit parsl.executors.status_handling INFO: Allocated block ID 5857 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5857 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5857 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5857 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5857 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5857.1709794017.915614.submit parsl.executors.status_handling INFO: Allocated block ID 5858 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5858 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5858 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5858 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5858 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5858.1709794017.9160426.submit parsl.executors.status_handling INFO: Allocated block ID 5859 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5859 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5859 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5859 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5859 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5859.1709794017.9164631.submit parsl.executors.status_handling INFO: Allocated block ID 5860 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5860 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5860 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5860 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5860 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5860.1709794017.9169257.submit parsl.executors.status_handling INFO: Allocated block ID 5861 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5861 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5861 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5861 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5861 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5861.1709794017.9173522.submit parsl.executors.status_handling INFO: Allocated block ID 5862 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5862 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5862 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5862 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5862 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5862.1709794017.9177685.submit parsl.executors.status_handling INFO: Allocated block ID 5863 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5863 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5863 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5863 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5863 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5863.1709794017.9182093.submit parsl.executors.status_handling INFO: Allocated block ID 5864 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5864 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5864 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5864 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5864 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5864.1709794017.9186292.submit parsl.executors.status_handling INFO: Allocated block ID 5865 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5865 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5865 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5865 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5865 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5865.1709794017.9190493.submit parsl.executors.status_handling INFO: Allocated block ID 5866 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5866 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5866 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5866 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5866 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5866.1709794017.919467.submit parsl.executors.status_handling INFO: Allocated block ID 5867 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5867 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5867 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5867 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5867 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5867.1709794017.9198923.submit parsl.executors.status_handling INFO: Allocated block ID 5868 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5868 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5868 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5868 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5868 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5868.1709794017.9203143.submit parsl.executors.status_handling INFO: Allocated block ID 5869 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5869 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5869 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5869 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5869 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5869.1709794017.9207287.submit parsl.executors.status_handling INFO: Allocated block ID 5870 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5870 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5870 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5870 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5870 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5870.1709794017.921151.submit parsl.executors.status_handling INFO: Allocated block ID 5871 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5871 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5871 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5871 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5871 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5871.1709794017.9215665.submit parsl.executors.status_handling INFO: Allocated block ID 5872 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5872 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5872 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5872 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5872 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5872.1709794017.9219975.submit parsl.executors.status_handling INFO: Allocated block ID 5873 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5873 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5873 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5873 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5873 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5873.1709794017.922416.submit parsl.executors.status_handling INFO: Allocated block ID 5874 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5874 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5874 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5874 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5874 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5874.1709794017.922852.submit parsl.executors.status_handling INFO: Allocated block ID 5875 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5875 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5875 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5875 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5875 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5875.1709794017.9232798.submit parsl.executors.status_handling INFO: Allocated block ID 5876 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5876 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5876 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5876 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5876 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5876.1709794017.9237027.submit parsl.executors.status_handling INFO: Allocated block ID 5877 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5877 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5877 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5877 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5877 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5877.1709794017.924158.submit parsl.executors.status_handling INFO: Allocated block ID 5878 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5878 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5878 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5878 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5878 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5878.1709794017.9245822.submit parsl.executors.status_handling INFO: Allocated block ID 5879 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5879 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5879 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5879 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5879 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5879.1709794017.9351761.submit parsl.executors.status_handling INFO: Allocated block ID 5880 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5880 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5880 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5880 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5880 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5880.1709794017.935631.submit parsl.executors.status_handling INFO: Allocated block ID 5881 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5881 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5881 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5881 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5881 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5881.1709794017.9360688.submit parsl.executors.status_handling INFO: Allocated block ID 5882 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5882 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5882 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5882 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5882 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5882.1709794017.9364886.submit parsl.executors.status_handling INFO: Allocated block ID 5883 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5883 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5883 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5883 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5883 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5883.1709794017.936912.submit parsl.executors.status_handling INFO: Allocated block ID 5884 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5884 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5884 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5884 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5884 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5884.1709794017.9373474.submit parsl.executors.status_handling INFO: Allocated block ID 5885 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5885 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5885 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5885 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5885 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5885.1709794017.9377708.submit parsl.executors.status_handling INFO: Allocated block ID 5886 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5886 2024-03-06 22:46:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5886 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5886 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:46:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:46:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:46:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:46:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5886 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5886.1709794017.9381952.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1335 parsl.dataflow.strategy DEBUG: Executor multi has 1335 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 5887 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5887 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5887 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5887 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5887 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5887.1709794022.8124473.submit parsl.executors.status_handling INFO: Allocated block ID 5888 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5888 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5888 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5888 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5888 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5888.1709794022.8129528.submit parsl.executors.status_handling INFO: Allocated block ID 5889 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5889 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5889 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5889 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5889 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5889.1709794022.8133948.submit parsl.executors.status_handling INFO: Allocated block ID 5890 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5890 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5890 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5890 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5890 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5890.1709794022.8138275.submit parsl.executors.status_handling INFO: Allocated block ID 5891 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5891 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5891 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5891 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5891 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5891.1709794022.8142543.submit parsl.executors.status_handling INFO: Allocated block ID 5892 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5892 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5892 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5892 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5892 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5892.1709794022.8146818.submit parsl.executors.status_handling INFO: Allocated block ID 5893 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5893 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5893 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5893 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5893 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5893.1709794022.8151078.submit parsl.executors.status_handling INFO: Allocated block ID 5894 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5894 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5894 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5894 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5894 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5894.1709794022.815541.submit parsl.executors.status_handling INFO: Allocated block ID 5895 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5895 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5895 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5895 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5895 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5895.1709794022.8159719.submit parsl.executors.status_handling INFO: Allocated block ID 5896 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5896 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5896 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5896 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5896 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5896.1709794022.816395.submit parsl.executors.status_handling INFO: Allocated block ID 5897 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5897 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5897 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5897 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5897 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5897.1709794022.816819.submit parsl.executors.status_handling INFO: Allocated block ID 5898 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5898 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5898 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5898 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5898 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5898.1709794022.817247.submit parsl.executors.status_handling INFO: Allocated block ID 5899 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5899 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5899 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5899 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5899 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5899.1709794022.8176677.submit parsl.executors.status_handling INFO: Allocated block ID 5900 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5900 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5900 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5900 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5900 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5900.1709794022.8181.submit parsl.executors.status_handling INFO: Allocated block ID 5901 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5901 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5901 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5901 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5901 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5901.1709794022.81852.submit parsl.executors.status_handling INFO: Allocated block ID 5902 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5902 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5902 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5902 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5902 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5902.1709794022.818943.submit parsl.executors.status_handling INFO: Allocated block ID 5903 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5903 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5903 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5903 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5903 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5903.1709794022.8193622.submit parsl.executors.status_handling INFO: Allocated block ID 5904 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5904 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5904 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5904 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5904 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5904.1709794022.819776.submit parsl.executors.status_handling INFO: Allocated block ID 5905 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5905 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5905 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5905 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5905 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5905.1709794022.8202014.submit parsl.executors.status_handling INFO: Allocated block ID 5906 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5906 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5906 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5906 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5906 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5906.1709794022.820631.submit parsl.executors.status_handling INFO: Allocated block ID 5907 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5907 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5907 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5907 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5907 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5907.1709794022.8210647.submit parsl.executors.status_handling INFO: Allocated block ID 5908 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5908 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5908 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5908 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5908 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5908.1709794022.8214872.submit parsl.executors.status_handling INFO: Allocated block ID 5909 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5909 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5909 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5909 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5909 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5909.1709794022.8219116.submit parsl.executors.status_handling INFO: Allocated block ID 5910 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5910 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5910 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5910 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5910 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5910.1709794022.822332.submit parsl.executors.status_handling INFO: Allocated block ID 5911 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5911 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5911 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5911 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5911 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5911.1709794022.8227482.submit parsl.executors.status_handling INFO: Allocated block ID 5912 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5912 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5912 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5912 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5912 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5912.1709794022.823174.submit parsl.executors.status_handling INFO: Allocated block ID 5913 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5913 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5913 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5913 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5913 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5913.1709794022.8235896.submit parsl.executors.status_handling INFO: Allocated block ID 5914 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5914 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5914 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5914 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5914 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5914.1709794022.8240125.submit parsl.executors.status_handling INFO: Allocated block ID 5915 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5915 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5915 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5915 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5915 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5915.1709794022.8244286.submit parsl.executors.status_handling INFO: Allocated block ID 5916 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5916 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5916 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5916 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5916 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5916.1709794022.8248494.submit parsl.executors.status_handling INFO: Allocated block ID 5917 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5917 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5917 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5917 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5917 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5917.1709794022.8252661.submit parsl.executors.status_handling INFO: Allocated block ID 5918 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5918 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5918 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5918 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5918 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5918.1709794022.8256938.submit parsl.executors.status_handling INFO: Allocated block ID 5919 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5919 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5919 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5919 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5919 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5919.1709794022.8261328.submit parsl.executors.status_handling INFO: Allocated block ID 5920 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5920 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5920 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5920 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5920 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5920.1709794022.8265543.submit parsl.executors.status_handling INFO: Allocated block ID 5921 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5921 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5921 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5921 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5921 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5921.1709794022.8269806.submit parsl.executors.status_handling INFO: Allocated block ID 5922 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5922 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5922 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5922 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5922 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5922.1709794022.8274007.submit parsl.executors.status_handling INFO: Allocated block ID 5923 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5923 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5923 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5923 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5923 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5923.1709794022.827833.submit parsl.executors.status_handling INFO: Allocated block ID 5924 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5924 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5924 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5924 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5924 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5924.1709794022.8282557.submit parsl.executors.status_handling INFO: Allocated block ID 5925 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5925 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5925 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5925 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5925 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5925.1709794022.828671.submit parsl.executors.status_handling INFO: Allocated block ID 5926 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5926 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5926 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5926 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5926 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5926.1709794022.829092.submit parsl.executors.status_handling INFO: Allocated block ID 5927 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5927 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5927 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5927 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5927 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5927.1709794022.8295052.submit parsl.executors.status_handling INFO: Allocated block ID 5928 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5928 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5928 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5928 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5928 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5928.1709794022.82993.submit parsl.executors.status_handling INFO: Allocated block ID 5929 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5929 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5929 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5929 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5929 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5929.1709794022.830347.submit parsl.executors.status_handling INFO: Allocated block ID 5930 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5930 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5930 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5930 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5930 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5930.1709794022.830771.submit parsl.executors.status_handling INFO: Allocated block ID 5931 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5931 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5931 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5931 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5931 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5931.1709794022.8312016.submit parsl.executors.status_handling INFO: Allocated block ID 5932 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5932 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5932 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5932 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5932 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5932.1709794022.8316207.submit parsl.executors.status_handling INFO: Allocated block ID 5933 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5933 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5933 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5933 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5933 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5933.1709794022.8320425.submit parsl.executors.status_handling INFO: Allocated block ID 5934 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5934 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5934 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5934 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5934 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5934.1709794022.83246.submit parsl.executors.status_handling INFO: Allocated block ID 5935 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5935 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5935 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5935 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5935 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5935.1709794022.8459423.submit parsl.executors.status_handling INFO: Allocated block ID 5936 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5936 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5936 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5936 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5936 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5936.1709794022.8464015.submit parsl.executors.status_handling INFO: Allocated block ID 5937 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5937 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5937 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5937 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5937 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5937.1709794022.846834.submit parsl.executors.status_handling INFO: Allocated block ID 5938 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5938 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5938 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5938 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5938 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5938.1709794022.8472517.submit parsl.executors.status_handling INFO: Allocated block ID 5939 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5939 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5939 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5939 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5939 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5939.1709794022.8476694.submit parsl.executors.status_handling INFO: Allocated block ID 5940 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5940 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5940 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5940 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5940 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5940.1709794022.8480952.submit parsl.executors.status_handling INFO: Allocated block ID 5941 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5941 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5941 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5941 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5941 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5941.1709794022.848521.submit parsl.executors.status_handling INFO: Allocated block ID 5942 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5942 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5942 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5942 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5942 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5942.1709794022.8489602.submit parsl.executors.status_handling INFO: Allocated block ID 5943 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5943 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5943 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5943 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5943 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5943.1709794022.8493865.submit parsl.executors.status_handling INFO: Allocated block ID 5944 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5944 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5944 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5944 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5944 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5944.1709794022.8498132.submit parsl.executors.status_handling INFO: Allocated block ID 5945 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5945 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5945 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5945 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5945 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5945.1709794022.8502395.submit parsl.executors.status_handling INFO: Allocated block ID 5946 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5946 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5946 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5946 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5946 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5946.1709794022.850659.submit parsl.executors.status_handling INFO: Allocated block ID 5947 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5947 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5947 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5947 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5947 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5947.1709794022.8510833.submit parsl.executors.status_handling INFO: Allocated block ID 5948 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5948 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5948 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5948 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5948 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5948.1709794022.8515065.submit parsl.executors.status_handling INFO: Allocated block ID 5949 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5949 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5949 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5949 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5949 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5949.1709794022.8519309.submit parsl.executors.status_handling INFO: Allocated block ID 5950 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5950 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5950 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5950 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5950 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5950.1709794022.852347.submit parsl.executors.status_handling INFO: Allocated block ID 5951 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5951 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5951 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5951 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5951 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5951.1709794022.8527644.submit parsl.executors.status_handling INFO: Allocated block ID 5952 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5952 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5952 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5952 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5952 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5952.1709794022.8531957.submit parsl.executors.status_handling INFO: Allocated block ID 5953 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5953 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5953 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5953 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5953 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5953.1709794022.8536134.submit parsl.executors.status_handling INFO: Allocated block ID 5954 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5954 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5954 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5954 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5954 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5954.1709794022.8540444.submit parsl.executors.status_handling INFO: Allocated block ID 5955 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5955 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5955 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5955 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5955 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5955.1709794022.8544803.submit parsl.executors.status_handling INFO: Allocated block ID 5956 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5956 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5956 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5956 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5956 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5956.1709794022.8549163.submit parsl.executors.status_handling INFO: Allocated block ID 5957 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5957 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5957 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5957 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5957 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5957.1709794022.855346.submit parsl.executors.status_handling INFO: Allocated block ID 5958 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5958 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5958 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5958 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5958 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5958.1709794022.8557837.submit parsl.executors.status_handling INFO: Allocated block ID 5959 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5959 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5959 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5959 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5959 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5959.1709794022.8562071.submit parsl.executors.status_handling INFO: Allocated block ID 5960 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5960 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5960 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5960 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5960 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5960.1709794022.856629.submit parsl.executors.status_handling INFO: Allocated block ID 5961 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5961 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5961 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5961 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5961 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5961.1709794022.857055.submit parsl.executors.status_handling INFO: Allocated block ID 5962 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5962 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5962 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5962 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5962 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5962.1709794022.857481.submit parsl.executors.status_handling INFO: Allocated block ID 5963 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5963 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5963 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5963 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5963 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5963.1709794022.857905.submit parsl.executors.status_handling INFO: Allocated block ID 5964 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5964 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5964 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5964 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5964 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5964.1709794022.8583224.submit parsl.executors.status_handling INFO: Allocated block ID 5965 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5965 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5965 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5965 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5965 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5965.1709794022.8587384.submit parsl.executors.status_handling INFO: Allocated block ID 5966 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5966 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5966 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5966 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5966 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5966.1709794022.872254.submit parsl.executors.status_handling INFO: Allocated block ID 5967 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5967 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5967 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5967 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5967 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5967.1709794022.8726883.submit parsl.executors.status_handling INFO: Allocated block ID 5968 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5968 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5968 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5968 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5968 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5968.1709794022.8731222.submit parsl.executors.status_handling INFO: Allocated block ID 5969 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5969 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5969 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5969 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5969 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5969.1709794022.873539.submit parsl.executors.status_handling INFO: Allocated block ID 5970 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5970 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5970 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5970 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5970 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5970.1709794022.87396.submit parsl.executors.status_handling INFO: Allocated block ID 5971 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5971 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5971 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5971 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5971 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5971.1709794022.8743777.submit parsl.executors.status_handling INFO: Allocated block ID 5972 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5972 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5972 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5972 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5972 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5972.1709794022.8747945.submit parsl.executors.status_handling INFO: Allocated block ID 5973 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5973 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5973 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5973 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5973 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5973.1709794022.8752172.submit parsl.executors.status_handling INFO: Allocated block ID 5974 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5974 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5974 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5974 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5974 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5974.1709794022.8756332.submit parsl.executors.status_handling INFO: Allocated block ID 5975 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5975 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5975 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5975 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5975 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5975.1709794022.876055.submit parsl.executors.status_handling INFO: Allocated block ID 5976 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5976 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5976 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5976 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5976 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5976.1709794022.8764732.submit parsl.executors.status_handling INFO: Allocated block ID 5977 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5977 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5977 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5977 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5977 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5977.1709794022.876896.submit parsl.executors.status_handling INFO: Allocated block ID 5978 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5978 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5978 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5978 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5978 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5978.1709794022.8773248.submit parsl.executors.status_handling INFO: Allocated block ID 5979 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5979 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5979 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5979 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5979 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5979.1709794022.8777719.submit parsl.executors.status_handling INFO: Allocated block ID 5980 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5980 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5980 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5980 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5980 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5980.1709794022.8782048.submit parsl.executors.status_handling INFO: Allocated block ID 5981 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5981 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5981 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5981 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5981 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5981.1709794022.8786328.submit parsl.executors.status_handling INFO: Allocated block ID 5982 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5982 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5982 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5982 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5982 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5982.1709794022.879068.submit parsl.executors.status_handling INFO: Allocated block ID 5983 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5983 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5983 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5983 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5983 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5983.1709794022.8794951.submit parsl.executors.status_handling INFO: Allocated block ID 5984 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5984 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5984 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5984 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5984 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5984.1709794022.8799434.submit parsl.executors.status_handling INFO: Allocated block ID 5985 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5985 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5985 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5985 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5985 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5985.1709794022.8803723.submit parsl.executors.status_handling INFO: Allocated block ID 5986 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5986 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5986 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5986 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5986 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5986.1709794022.8807974.submit parsl.executors.status_handling INFO: Allocated block ID 5987 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5987 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5987 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5987 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5987 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5987.1709794022.8812315.submit parsl.executors.status_handling INFO: Allocated block ID 5988 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5988 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5988 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5988 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5988 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5988.1709794022.8816557.submit parsl.executors.status_handling INFO: Allocated block ID 5989 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5989 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5989 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5989 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5989 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5989.1709794022.882087.submit parsl.executors.status_handling INFO: Allocated block ID 5990 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5990 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5990 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5990 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5990 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5990.1709794022.8825228.submit parsl.executors.status_handling INFO: Allocated block ID 5991 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5991 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5991 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5991 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5991 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5991.1709794022.882959.submit parsl.executors.status_handling INFO: Allocated block ID 5992 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5992 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5992 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5992 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5992 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5992.1709794022.8833854.submit parsl.executors.status_handling INFO: Allocated block ID 5993 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5993 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5993 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5993 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5993 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5993.1709794022.8837988.submit parsl.executors.status_handling INFO: Allocated block ID 5994 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5994 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5994 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5994 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5994 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5994.1709794022.8842309.submit parsl.executors.status_handling INFO: Allocated block ID 5995 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5995 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5995 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5995 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5995 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5995.1709794022.884648.submit parsl.executors.status_handling INFO: Allocated block ID 5996 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5996 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5996 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5996 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5996 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5996.1709794022.898165.submit parsl.executors.status_handling INFO: Allocated block ID 5997 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5997 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5997 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5997 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5997 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5997.1709794022.8986087.submit parsl.executors.status_handling INFO: Allocated block ID 5998 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5998 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5998 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5998 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5998 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5998.1709794022.8990583.submit parsl.executors.status_handling INFO: Allocated block ID 5999 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5999 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5999 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5999 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=5999 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-5999.1709794022.899487.submit parsl.executors.status_handling INFO: Allocated block ID 6000 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6000 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6000 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6000 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6000 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6000.1709794022.8999221.submit parsl.executors.status_handling INFO: Allocated block ID 6001 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6001 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6001 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6001 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6001 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6001.1709794022.900355.submit parsl.executors.status_handling INFO: Allocated block ID 6002 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6002 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6002 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6002 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6002 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6002.1709794022.9007847.submit parsl.executors.status_handling INFO: Allocated block ID 6003 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6003 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6003 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6003 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6003 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6003.1709794022.9012375.submit parsl.executors.status_handling INFO: Allocated block ID 6004 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6004 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6004 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6004 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6004 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6004.1709794022.9016697.submit parsl.executors.status_handling INFO: Allocated block ID 6005 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6005 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6005 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6005 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6005 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6005.1709794022.9021056.submit parsl.executors.status_handling INFO: Allocated block ID 6006 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6006 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6006 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6006 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6006 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6006.1709794022.9025333.submit parsl.executors.status_handling INFO: Allocated block ID 6007 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6007 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6007 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6007 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6007 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6007.1709794022.9029686.submit parsl.executors.status_handling INFO: Allocated block ID 6008 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6008 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6008 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6008 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6008 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6008.1709794022.9034004.submit parsl.executors.status_handling INFO: Allocated block ID 6009 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6009 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6009 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6009 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6009 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6009.1709794022.9038217.submit parsl.executors.status_handling INFO: Allocated block ID 6010 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6010 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6010 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6010 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6010 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6010.1709794022.9042654.submit parsl.executors.status_handling INFO: Allocated block ID 6011 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6011 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6011 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6011 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6011 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6011.1709794022.9046924.submit parsl.executors.status_handling INFO: Allocated block ID 6012 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6012 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6012 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6012 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6012 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6012.1709794022.905142.submit parsl.executors.status_handling INFO: Allocated block ID 6013 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6013 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6013 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6013 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6013 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6013.1709794022.9055634.submit parsl.executors.status_handling INFO: Allocated block ID 6014 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6014 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6014 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6014 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6014 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6014.1709794022.9059932.submit parsl.executors.status_handling INFO: Allocated block ID 6015 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6015 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6015 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6015 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6015 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6015.1709794022.9064133.submit parsl.executors.status_handling INFO: Allocated block ID 6016 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6016 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6016 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6016 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6016 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6016.1709794022.9068391.submit parsl.executors.status_handling INFO: Allocated block ID 6017 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6017 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6017 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6017 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6017 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6017.1709794022.9072566.submit parsl.executors.status_handling INFO: Allocated block ID 6018 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6018 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6018 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6018 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6018 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6018.1709794022.907675.submit parsl.executors.status_handling INFO: Allocated block ID 6019 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6019 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6019 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6019 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6019 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6019.1709794022.9080994.submit parsl.executors.status_handling INFO: Allocated block ID 6020 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6020 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6020 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6020 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6020 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6020.1709794022.9085183.submit parsl.executors.status_handling INFO: Allocated block ID 6021 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6021 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6021 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6021 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6021 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6021.1709794022.9089396.submit parsl.executors.status_handling INFO: Allocated block ID 6022 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6022 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6022 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6022 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6022 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6022.1709794022.9093602.submit parsl.executors.status_handling INFO: Allocated block ID 6023 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6023 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6023 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6023 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6023 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6023.1709794022.909775.submit parsl.executors.status_handling INFO: Allocated block ID 6024 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6024 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6024 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6024 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6024 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6024.1709794022.9101975.submit parsl.executors.status_handling INFO: Allocated block ID 6025 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6025 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6025 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6025 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6025 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6025.1709794022.910611.submit parsl.executors.status_handling INFO: Allocated block ID 6026 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6026 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6026 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6026 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6026 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6026.1709794022.9241805.submit parsl.executors.status_handling INFO: Allocated block ID 6027 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6027 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6027 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6027 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6027 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6027.1709794022.9246554.submit parsl.executors.status_handling INFO: Allocated block ID 6028 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6028 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6028 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6028 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6028 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6028.1709794022.9250963.submit parsl.executors.status_handling INFO: Allocated block ID 6029 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6029 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6029 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6029 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6029 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6029.1709794022.9255204.submit parsl.executors.status_handling INFO: Allocated block ID 6030 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6030 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6030 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6030 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6030 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6030.1709794022.9259474.submit parsl.executors.status_handling INFO: Allocated block ID 6031 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6031 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6031 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6031 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6031 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6031.1709794022.9263704.submit parsl.executors.status_handling INFO: Allocated block ID 6032 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6032 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6032 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6032 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6032 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6032.1709794022.9267936.submit parsl.executors.status_handling INFO: Allocated block ID 6033 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6033 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6033 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6033 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6033 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6033.1709794022.9272187.submit parsl.executors.status_handling INFO: Allocated block ID 6034 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6034 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6034 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6034 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6034 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6034.1709794022.9276338.submit parsl.executors.status_handling INFO: Allocated block ID 6035 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6035 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6035 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6035 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6035 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6035.1709794022.9280827.submit parsl.executors.status_handling INFO: Allocated block ID 6036 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6036 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6036 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6036 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6036 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6036.1709794022.9285104.submit parsl.executors.status_handling INFO: Allocated block ID 6037 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6037 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6037 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6037 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6037 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6037.1709794022.9289432.submit parsl.executors.status_handling INFO: Allocated block ID 6038 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6038 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6038 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6038 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6038 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6038.1709794022.9293833.submit parsl.executors.status_handling INFO: Allocated block ID 6039 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6039 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6039 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6039 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6039 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6039.1709794022.9298017.submit parsl.executors.status_handling INFO: Allocated block ID 6040 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6040 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6040 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6040 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6040 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6040.1709794022.930225.submit parsl.executors.status_handling INFO: Allocated block ID 6041 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6041 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6041 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6041 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6041 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6041.1709794022.930641.submit parsl.executors.status_handling INFO: Allocated block ID 6042 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6042 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6042 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6042 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6042 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6042.1709794022.9310656.submit parsl.executors.status_handling INFO: Allocated block ID 6043 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6043 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6043 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6043 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6043 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6043.1709794022.9314852.submit parsl.executors.status_handling INFO: Allocated block ID 6044 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6044 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6044 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6044 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6044 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6044.1709794022.9319167.submit parsl.executors.status_handling INFO: Allocated block ID 6045 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6045 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6045 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6045 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6045 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6045.1709794022.9323354.submit parsl.executors.status_handling INFO: Allocated block ID 6046 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6046 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6046 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6046 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6046 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6046.1709794022.9327507.submit parsl.executors.status_handling INFO: Allocated block ID 6047 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6047 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6047 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6047 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6047 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6047.1709794022.9331791.submit parsl.executors.status_handling INFO: Allocated block ID 6048 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6048 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6048 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6048 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6048 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6048.1709794022.9335935.submit parsl.executors.status_handling INFO: Allocated block ID 6049 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6049 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6049 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6049 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6049 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6049.1709794022.934034.submit parsl.executors.status_handling INFO: Allocated block ID 6050 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6050 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6050 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6050 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6050 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6050.1709794022.9344752.submit parsl.executors.status_handling INFO: Allocated block ID 6051 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6051 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6051 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6051 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6051 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6051.1709794022.9349124.submit parsl.executors.status_handling INFO: Allocated block ID 6052 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6052 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6052 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6052 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6052 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6052.1709794022.9353478.submit parsl.executors.status_handling INFO: Allocated block ID 6053 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6053 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6053 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6053 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6053 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6053.1709794022.9357665.submit parsl.executors.status_handling INFO: Allocated block ID 6054 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6054 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6054 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6054 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6054 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6054.1709794022.9361897.submit parsl.executors.status_handling INFO: Allocated block ID 6055 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6055 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6055 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6055 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6055 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6055.1709794022.9366097.submit parsl.executors.status_handling INFO: Allocated block ID 6056 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6056 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6056 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6056 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6056 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6056.1709794022.9423835.submit parsl.executors.status_handling INFO: Allocated block ID 6057 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6057 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6057 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6057 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6057 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6057.1709794022.942866.submit parsl.executors.status_handling INFO: Allocated block ID 6058 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6058 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6058 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6058 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6058 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6058.1709794022.9433122.submit parsl.executors.status_handling INFO: Allocated block ID 6059 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6059 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6059 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6059 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6059 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6059.1709794022.9437332.submit parsl.executors.status_handling INFO: Allocated block ID 6060 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6060 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6060 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6060 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6060 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6060.1709794022.9467816.submit parsl.executors.status_handling INFO: Allocated block ID 6061 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6061 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6061 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6061 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6061 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6061.1709794022.9472506.submit parsl.executors.status_handling INFO: Allocated block ID 6062 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6062 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6062 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6062 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6062 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6062.1709794022.9476874.submit parsl.executors.status_handling INFO: Allocated block ID 6063 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6063 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6063 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6063 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6063 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6063.1709794022.948144.submit parsl.executors.status_handling INFO: Allocated block ID 6064 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6064 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6064 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6064 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6064 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6064.1709794022.948579.submit parsl.executors.status_handling INFO: Allocated block ID 6065 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6065 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6065 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6065 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6065 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6065.1709794022.9490216.submit parsl.executors.status_handling INFO: Allocated block ID 6066 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6066 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6066 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6066 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6066 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6066.1709794022.9494574.submit parsl.executors.status_handling INFO: Allocated block ID 6067 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6067 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6067 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6067 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6067 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6067.1709794022.9498847.submit parsl.executors.status_handling INFO: Allocated block ID 6068 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6068 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6068 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6068 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6068 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6068.1709794022.950311.submit parsl.executors.status_handling INFO: Allocated block ID 6069 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6069 2024-03-06 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6069 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6069 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6069 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6069.1709794022.9507275.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57a57b0c214b700cf74c57c90131035 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aa27dfca-e90d-4579-b6a8-a5af37caf47b_biasIsr_1123738_8.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123738/aa27dfca-e90d-4579-b6a8-a5af37caf47b_biasIsr_1123738_8.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae08f173c446c6ace811ae7fa88e1c9 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1485d868-1a9c-4c0b-9e8c-08790b7c8920_biasIsr_1123833_8.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123833/1485d868-1a9c-4c0b-9e8c-08790b7c8920_biasIsr_1123833_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1333 parsl.dataflow.strategy DEBUG: Executor multi has 1333 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 6070 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6070 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6070 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6070 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6070 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6070.1709794027.6920989.submit parsl.executors.status_handling INFO: Allocated block ID 6071 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6071 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6071 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6071 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6071 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6071.1709794027.6928234.submit parsl.executors.status_handling INFO: Allocated block ID 6072 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6072 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6072 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6072 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6072 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6072.1709794027.693502.submit parsl.executors.status_handling INFO: Allocated block ID 6073 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6073 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6073 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6073 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6073 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6073.1709794027.6941679.submit parsl.executors.status_handling INFO: Allocated block ID 6074 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6074 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6074 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6074 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6074 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6074.1709794027.6948388.submit parsl.executors.status_handling INFO: Allocated block ID 6075 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6075 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6075 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6075 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6075 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6075.1709794027.6955028.submit parsl.executors.status_handling INFO: Allocated block ID 6076 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6076 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6076 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6076 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6076 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6076.1709794027.6961603.submit parsl.executors.status_handling INFO: Allocated block ID 6077 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6077 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6077 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6077 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6077 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6077.1709794027.696806.submit parsl.executors.status_handling INFO: Allocated block ID 6078 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6078 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6078 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6078 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6078 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6078.1709794027.6974516.submit parsl.executors.status_handling INFO: Allocated block ID 6079 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6079 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6079 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6079 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6079 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6079.1709794027.6980963.submit parsl.executors.status_handling INFO: Allocated block ID 6080 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6080 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6080 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6080 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6080 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6080.1709794027.6987314.submit parsl.executors.status_handling INFO: Allocated block ID 6081 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6081 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6081 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6081 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6081 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6081.1709794027.6993787.submit parsl.executors.status_handling INFO: Allocated block ID 6082 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6082 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6082 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6082 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6082 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6082.1709794027.7002025.submit parsl.executors.status_handling INFO: Allocated block ID 6083 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6083 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6083 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6083 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6083 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6083.1709794027.7008767.submit parsl.executors.status_handling INFO: Allocated block ID 6084 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6084 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6084 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6084 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6084 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6084.1709794027.7015247.submit parsl.executors.status_handling INFO: Allocated block ID 6085 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6085 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6085 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6085 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6085 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6085.1709794027.702126.submit parsl.executors.status_handling INFO: Allocated block ID 6086 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6086 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6086 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6086 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6086 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6086.1709794027.7025526.submit parsl.executors.status_handling INFO: Allocated block ID 6087 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6087 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6087 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6087 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6087 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6087.1709794027.7029855.submit parsl.executors.status_handling INFO: Allocated block ID 6088 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6088 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6088 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6088 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6088 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6088.1709794027.7034109.submit parsl.executors.status_handling INFO: Allocated block ID 6089 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6089 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6089 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6089 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6089 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6089.1709794027.7038364.submit parsl.executors.status_handling INFO: Allocated block ID 6090 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6090 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6090 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6090 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6090 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6090.1709794027.7042608.submit parsl.executors.status_handling INFO: Allocated block ID 6091 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6091 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6091 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6091 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6091 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6091.1709794027.70468.submit parsl.executors.status_handling INFO: Allocated block ID 6092 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6092 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6092 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6092 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6092 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6092.1709794027.7051024.submit parsl.executors.status_handling INFO: Allocated block ID 6093 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6093 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6093 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6093 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6093 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6093.1709794027.7055204.submit parsl.executors.status_handling INFO: Allocated block ID 6094 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6094 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6094 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6094 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6094 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6094.1709794027.7059462.submit parsl.executors.status_handling INFO: Allocated block ID 6095 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6095 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6095 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6095 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6095 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6095.1709794027.7063627.submit parsl.executors.status_handling INFO: Allocated block ID 6096 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6096 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6096 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6096 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6096 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6096.1709794027.706778.submit parsl.executors.status_handling INFO: Allocated block ID 6097 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6097 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6097 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6097 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6097 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6097.1709794027.707216.submit parsl.executors.status_handling INFO: Allocated block ID 6098 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6098 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6098 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6098 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6098 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6098.1709794027.7076344.submit parsl.executors.status_handling INFO: Allocated block ID 6099 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6099 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6099 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6099 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6099 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6099.1709794027.7080579.submit parsl.executors.status_handling INFO: Allocated block ID 6100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6100 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6100 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6100.1709794027.708479.submit parsl.executors.status_handling INFO: Allocated block ID 6101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6101 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6101 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6101.1709794027.7089078.submit parsl.executors.status_handling INFO: Allocated block ID 6102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6102 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6102 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6102.1709794027.709326.submit parsl.executors.status_handling INFO: Allocated block ID 6103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6103 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6103 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6103.1709794027.7097476.submit parsl.executors.status_handling INFO: Allocated block ID 6104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6104 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6104 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6104.1709794027.7101796.submit parsl.executors.status_handling INFO: Allocated block ID 6105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6105 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6105 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6105.1709794027.7106013.submit parsl.executors.status_handling INFO: Allocated block ID 6106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6106 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6106 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6106.1709794027.7111452.submit parsl.executors.status_handling INFO: Allocated block ID 6107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6107 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6107 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6107.1709794027.711582.submit parsl.executors.status_handling INFO: Allocated block ID 6108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6108 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6108 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6108.1709794027.7120097.submit parsl.executors.status_handling INFO: Allocated block ID 6109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6109 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6109 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6109.1709794027.7124434.submit parsl.executors.status_handling INFO: Allocated block ID 6110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6110 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6110 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6110.1709794027.7128832.submit parsl.executors.status_handling INFO: Allocated block ID 6111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6111 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6111 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6111.1709794027.7133203.submit parsl.executors.status_handling INFO: Allocated block ID 6112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6112 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6112 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6112.1709794027.7137558.submit parsl.executors.status_handling INFO: Allocated block ID 6113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6113 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6113 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6113.1709794027.7141855.submit parsl.executors.status_handling INFO: Allocated block ID 6114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6114 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6114 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6114.1709794027.7146041.submit parsl.executors.status_handling INFO: Allocated block ID 6115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6115 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6115 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6115.1709794027.7150328.submit parsl.executors.status_handling INFO: Allocated block ID 6116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6116 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6116 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6116.1709794027.7154732.submit parsl.executors.status_handling INFO: Allocated block ID 6117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6117 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6117 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6117.1709794027.7159202.submit parsl.executors.status_handling INFO: Allocated block ID 6118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6118 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6118 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6118.1709794027.7163596.submit parsl.executors.status_handling INFO: Allocated block ID 6119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6119 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6119 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6119.1709794027.7167928.submit parsl.executors.status_handling INFO: Allocated block ID 6120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6120 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6120 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6120.1709794027.7172294.submit parsl.executors.status_handling INFO: Allocated block ID 6121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6121 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6121 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6121.1709794027.717678.submit parsl.executors.status_handling INFO: Allocated block ID 6122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6122 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6122 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6122.1709794027.7181063.submit parsl.executors.status_handling INFO: Allocated block ID 6123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6123 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6123 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6123.1709794027.7185266.submit parsl.executors.status_handling INFO: Allocated block ID 6124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6124 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6124 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6124.1709794027.7189639.submit parsl.executors.status_handling INFO: Allocated block ID 6125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6125 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6125 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6125.1709794027.7193892.submit parsl.executors.status_handling INFO: Allocated block ID 6126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6126 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6126 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6126.1709794027.719814.submit parsl.executors.status_handling INFO: Allocated block ID 6127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6127 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6127 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6127.1709794027.7202601.submit parsl.executors.status_handling INFO: Allocated block ID 6128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6128 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6128 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6128.1709794027.7206893.submit parsl.executors.status_handling INFO: Allocated block ID 6129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6129 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6129 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6129.1709794027.721376.submit parsl.executors.status_handling INFO: Allocated block ID 6130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6130 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6130 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6130.1709794027.7218332.submit parsl.executors.status_handling INFO: Allocated block ID 6131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6131 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6131 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6131.1709794027.7222679.submit parsl.executors.status_handling INFO: Allocated block ID 6132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6132 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6132 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6132.1709794027.7227054.submit parsl.executors.status_handling INFO: Allocated block ID 6133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6133 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6133 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6133.1709794027.7231538.submit parsl.executors.status_handling INFO: Allocated block ID 6134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6134 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6134 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6134.1709794027.7235873.submit parsl.executors.status_handling INFO: Allocated block ID 6135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6135 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6135 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6135.1709794027.724023.submit parsl.executors.status_handling INFO: Allocated block ID 6136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6136 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6136 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6136.1709794027.7244544.submit parsl.executors.status_handling INFO: Allocated block ID 6137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6137 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6137 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6137.1709794027.7248757.submit parsl.executors.status_handling INFO: Allocated block ID 6138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6138 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6138 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6138.1709794027.7252908.submit parsl.executors.status_handling INFO: Allocated block ID 6139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6139 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6139 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6139.1709794027.7257116.submit parsl.executors.status_handling INFO: Allocated block ID 6140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6140 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6140 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6140.1709794027.7261436.submit parsl.executors.status_handling INFO: Allocated block ID 6141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6141 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6141 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6141.1709794027.726559.submit parsl.executors.status_handling INFO: Allocated block ID 6142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6142 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6142 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6142.1709794027.726992.submit parsl.executors.status_handling INFO: Allocated block ID 6143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6143 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6143 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6143.1709794027.7274113.submit parsl.executors.status_handling INFO: Allocated block ID 6144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6144 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6144 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6144.1709794027.7278364.submit parsl.executors.status_handling INFO: Allocated block ID 6145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6145 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6145 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6145.1709794027.728263.submit parsl.executors.status_handling INFO: Allocated block ID 6146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6146 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6146 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6146.1709794027.728702.submit parsl.executors.status_handling INFO: Allocated block ID 6147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6147 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6147 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6147.1709794027.729145.submit parsl.executors.status_handling INFO: Allocated block ID 6148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6148 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6148 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6148.1709794027.7295787.submit parsl.executors.status_handling INFO: Allocated block ID 6149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6149 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6149 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6149.1709794027.7300124.submit parsl.executors.status_handling INFO: Allocated block ID 6150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6150 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6150 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6150.1709794027.7304454.submit parsl.executors.status_handling INFO: Allocated block ID 6151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6151 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6151 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6151.1709794027.7308683.submit parsl.executors.status_handling INFO: Allocated block ID 6152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6152 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6152 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6152.1709794027.7313008.submit parsl.executors.status_handling INFO: Allocated block ID 6153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6153 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6153 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6153.1709794027.7317235.submit parsl.executors.status_handling INFO: Allocated block ID 6154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6154 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6154 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6154.1709794027.7323122.submit parsl.executors.status_handling INFO: Allocated block ID 6155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6155 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6155 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6155.1709794027.7327406.submit parsl.executors.status_handling INFO: Allocated block ID 6156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6156 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6156 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6156.1709794027.7331703.submit parsl.executors.status_handling INFO: Allocated block ID 6157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6157 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6157 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6157.1709794027.733602.submit parsl.executors.status_handling INFO: Allocated block ID 6158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6158 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6158 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6158.1709794027.7340314.submit parsl.executors.status_handling INFO: Allocated block ID 6159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6159 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6159 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6159.1709794027.7344737.submit parsl.executors.status_handling INFO: Allocated block ID 6160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6160 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6160 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6160.1709794027.7349184.submit parsl.executors.status_handling INFO: Allocated block ID 6161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6161 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6161 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6161.1709794027.7353501.submit parsl.executors.status_handling INFO: Allocated block ID 6162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6162 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6162 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6162.1709794027.7357821.submit parsl.executors.status_handling INFO: Allocated block ID 6163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6163 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6163 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6163.1709794027.73621.submit parsl.executors.status_handling INFO: Allocated block ID 6164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6164 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6164 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6164.1709794027.7366266.submit parsl.executors.status_handling INFO: Allocated block ID 6165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6165 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6165 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6165.1709794027.7370508.submit parsl.executors.status_handling INFO: Allocated block ID 6166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6166 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6166 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6166.1709794027.7374759.submit parsl.executors.status_handling INFO: Allocated block ID 6167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6167 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6167 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6167.1709794027.7378979.submit parsl.executors.status_handling INFO: Allocated block ID 6168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6168 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6168 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6168.1709794027.7383175.submit parsl.executors.status_handling INFO: Allocated block ID 6169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6169 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6169 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6169.1709794027.73874.submit parsl.executors.status_handling INFO: Allocated block ID 6170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6170 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6170 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6170.1709794027.7391703.submit parsl.executors.status_handling INFO: Allocated block ID 6171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6171 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6171 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6171.1709794027.7395844.submit parsl.executors.status_handling INFO: Allocated block ID 6172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6172 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6172 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6172.1709794027.740009.submit parsl.executors.status_handling INFO: Allocated block ID 6173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6173 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6173 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6173.1709794027.7404258.submit parsl.executors.status_handling INFO: Allocated block ID 6174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6174 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6174 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6174.1709794027.7408512.submit parsl.executors.status_handling INFO: Allocated block ID 6175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6175 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6175 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6175.1709794027.7414162.submit parsl.executors.status_handling INFO: Allocated block ID 6176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6176 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6176 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6176.1709794027.741845.submit parsl.executors.status_handling INFO: Allocated block ID 6177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6177 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6177 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6177.1709794027.7430387.submit parsl.executors.status_handling INFO: Allocated block ID 6178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6178 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6178 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6178.1709794027.7434726.submit parsl.executors.status_handling INFO: Allocated block ID 6179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6179 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6179 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6179.1709794027.7439058.submit parsl.executors.status_handling INFO: Allocated block ID 6180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6180 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6180 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6180.1709794027.7443287.submit parsl.executors.status_handling INFO: Allocated block ID 6181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6181 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6181 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6181.1709794027.7447567.submit parsl.executors.status_handling INFO: Allocated block ID 6182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6182 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6182 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6182.1709794027.7451909.submit parsl.executors.status_handling INFO: Allocated block ID 6183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6183 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6183 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6183.1709794027.7456145.submit parsl.executors.status_handling INFO: Allocated block ID 6184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6184 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6184 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6184.1709794027.7460487.submit parsl.executors.status_handling INFO: Allocated block ID 6185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6185 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6185 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6185.1709794027.746468.submit parsl.executors.status_handling INFO: Allocated block ID 6186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6186 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6186 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6186.1709794027.746896.submit parsl.executors.status_handling INFO: Allocated block ID 6187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6187 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6187 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6187.1709794027.7473137.submit parsl.executors.status_handling INFO: Allocated block ID 6188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6188 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6188 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6188.1709794027.7477272.submit parsl.executors.status_handling INFO: Allocated block ID 6189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6189 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6189 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6189.1709794027.7481472.submit parsl.executors.status_handling INFO: Allocated block ID 6190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6190 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6190 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6190.1709794027.7485673.submit parsl.executors.status_handling INFO: Allocated block ID 6191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6191 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6191 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6191.1709794027.7489924.submit parsl.executors.status_handling INFO: Allocated block ID 6192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6192 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6192 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6192.1709794027.7494142.submit parsl.executors.status_handling INFO: Allocated block ID 6193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6193 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6193 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6193.1709794027.7498465.submit parsl.executors.status_handling INFO: Allocated block ID 6194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6194 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6194 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6194.1709794027.7502685.submit parsl.executors.status_handling INFO: Allocated block ID 6195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6195 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6195 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6195.1709794027.75069.submit parsl.executors.status_handling INFO: Allocated block ID 6196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6196 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6196 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6196.1709794027.7511327.submit parsl.executors.status_handling INFO: Allocated block ID 6197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6197 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6197 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6197.1709794027.7515564.submit parsl.executors.status_handling INFO: Allocated block ID 6198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6198 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6198 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6198.1709794027.7519934.submit parsl.executors.status_handling INFO: Allocated block ID 6199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6199 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6199 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6199.1709794027.752416.submit parsl.executors.status_handling INFO: Allocated block ID 6200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6200 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6200 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6200.1709794027.7529614.submit parsl.executors.status_handling INFO: Allocated block ID 6201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6201 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6201 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6201.1709794027.7534068.submit parsl.executors.status_handling INFO: Allocated block ID 6202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6202 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6202 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6202.1709794027.75384.submit parsl.executors.status_handling INFO: Allocated block ID 6203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6203 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6203 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6203.1709794027.7542624.submit parsl.executors.status_handling INFO: Allocated block ID 6204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6204 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6204 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6204.1709794027.754678.submit parsl.executors.status_handling INFO: Allocated block ID 6205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6205 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6205 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6205.1709794027.7551131.submit parsl.executors.status_handling INFO: Allocated block ID 6206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6206 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6206 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6206.1709794027.75554.submit parsl.executors.status_handling INFO: Allocated block ID 6207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6207 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6207 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6207.1709794027.7559717.submit parsl.executors.status_handling INFO: Allocated block ID 6208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6208 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6208 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6208.1709794027.7563956.submit parsl.executors.status_handling INFO: Allocated block ID 6209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6209 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6209 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6209.1709794027.7568166.submit parsl.executors.status_handling INFO: Allocated block ID 6210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6210 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6210 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6210.1709794027.7572384.submit parsl.executors.status_handling INFO: Allocated block ID 6211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6211 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6211 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6211.1709794027.7576559.submit parsl.executors.status_handling INFO: Allocated block ID 6212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6212 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6212 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6212.1709794027.7580802.submit parsl.executors.status_handling INFO: Allocated block ID 6213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6213 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6213 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6213.1709794027.758501.submit parsl.executors.status_handling INFO: Allocated block ID 6214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6214 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6214 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6214.1709794027.7589307.submit parsl.executors.status_handling INFO: Allocated block ID 6215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6215 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6215 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6215.1709794027.75935.submit parsl.executors.status_handling INFO: Allocated block ID 6216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6216 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6216 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6216.1709794027.7597668.submit parsl.executors.status_handling INFO: Allocated block ID 6217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6217 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6217 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6217.1709794027.7602003.submit parsl.executors.status_handling INFO: Allocated block ID 6218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6218 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6218 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6218.1709794027.760644.submit parsl.executors.status_handling INFO: Allocated block ID 6219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6219 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6219 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6219.1709794027.7610784.submit parsl.executors.status_handling INFO: Allocated block ID 6220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6220 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6220 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6220.1709794027.761512.submit parsl.executors.status_handling INFO: Allocated block ID 6221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6221 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6221 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6221.1709794027.761944.submit parsl.executors.status_handling INFO: Allocated block ID 6222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6222 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6222 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6222.1709794027.7623646.submit parsl.executors.status_handling INFO: Allocated block ID 6223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6223 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6223 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6223.1709794027.7627857.submit parsl.executors.status_handling INFO: Allocated block ID 6224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6224 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6224 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6224.1709794027.763226.submit parsl.executors.status_handling INFO: Allocated block ID 6225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6225 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6225 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6225.1709794027.763642.submit parsl.executors.status_handling INFO: Allocated block ID 6226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6226 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6226 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6226.1709794027.7642205.submit parsl.executors.status_handling INFO: Allocated block ID 6227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6227 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6227 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6227.1709794027.7646694.submit parsl.executors.status_handling INFO: Allocated block ID 6228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6228 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6228 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6228.1709794027.7651093.submit parsl.executors.status_handling INFO: Allocated block ID 6229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6229 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6229 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6229.1709794027.7655487.submit parsl.executors.status_handling INFO: Allocated block ID 6230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6230 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6230 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6230.1709794027.765975.submit parsl.executors.status_handling INFO: Allocated block ID 6231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6231 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6231 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6231.1709794027.7663975.submit parsl.executors.status_handling INFO: Allocated block ID 6232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6232 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6232 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6232.1709794027.7668276.submit parsl.executors.status_handling INFO: Allocated block ID 6233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6233 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6233 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6233.1709794027.7672482.submit parsl.executors.status_handling INFO: Allocated block ID 6234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6234 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6234 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6234.1709794027.767665.submit parsl.executors.status_handling INFO: Allocated block ID 6235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6235 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6235 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6235.1709794027.7680914.submit parsl.executors.status_handling INFO: Allocated block ID 6236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6236 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6236 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6236.1709794027.7685087.submit parsl.executors.status_handling INFO: Allocated block ID 6237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6237 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6237 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6237.1709794027.7689555.submit parsl.executors.status_handling INFO: Allocated block ID 6238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6238 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6238 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6238.1709794027.769395.submit parsl.executors.status_handling INFO: Allocated block ID 6239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6239 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6239 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6239.1709794027.7698288.submit parsl.executors.status_handling INFO: Allocated block ID 6240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6240 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6240 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6240.1709794027.770263.submit parsl.executors.status_handling INFO: Allocated block ID 6241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6241 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6241 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6241.1709794027.7706854.submit parsl.executors.status_handling INFO: Allocated block ID 6242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6242 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6242 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6242.1709794027.7711153.submit parsl.executors.status_handling INFO: Allocated block ID 6243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6243 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6243 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6243.1709794027.7715383.submit parsl.executors.status_handling INFO: Allocated block ID 6244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6244 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6244 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6244.1709794027.771966.submit parsl.executors.status_handling INFO: Allocated block ID 6245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6245 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6245 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6245.1709794027.772387.submit parsl.executors.status_handling INFO: Allocated block ID 6246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6246 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6246 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6246.1709794027.7728033.submit parsl.executors.status_handling INFO: Allocated block ID 6247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6247 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6247 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6247.1709794027.7732582.submit parsl.executors.status_handling INFO: Allocated block ID 6248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6248 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6248 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6248.1709794027.773675.submit parsl.executors.status_handling INFO: Allocated block ID 6249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6249 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6249 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6249.1709794027.774312.submit parsl.executors.status_handling INFO: Allocated block ID 6250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6250 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6250 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6250.1709794027.7747686.submit parsl.executors.status_handling INFO: Allocated block ID 6251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6251 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6251 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6251.1709794027.775215.submit parsl.executors.status_handling INFO: Allocated block ID 6252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6252 2024-03-06 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6252 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6252.1709794027.775658.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd34826d8604e24b4047cae8b404400 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/158d77b0-5760-4708-ab77-adec3d41d89d_biasIsr_1123733_8.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123733/158d77b0-5760-4708-ab77-adec3d41d89d_biasIsr_1123733_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1332 parsl.dataflow.strategy DEBUG: Executor multi has 1332 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 6253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6253 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6253 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6253.1709794032.691759.submit parsl.executors.status_handling INFO: Allocated block ID 6254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6254 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6254 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6254.1709794032.6922922.submit parsl.executors.status_handling INFO: Allocated block ID 6255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6255 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6255 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6255.1709794032.6927562.submit parsl.executors.status_handling INFO: Allocated block ID 6256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6256 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6256 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6256.1709794032.6932166.submit parsl.executors.status_handling INFO: Allocated block ID 6257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6257 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6257 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6257.1709794032.6936648.submit parsl.executors.status_handling INFO: Allocated block ID 6258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6258 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6258 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6258.1709794032.6941104.submit parsl.executors.status_handling INFO: Allocated block ID 6259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6259 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6259 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6259.1709794032.694555.submit parsl.executors.status_handling INFO: Allocated block ID 6260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6260 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6260 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6260.1709794032.6949952.submit parsl.executors.status_handling INFO: Allocated block ID 6261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6261 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6261 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6261.1709794032.6954284.submit parsl.executors.status_handling INFO: Allocated block ID 6262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6262 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6262 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6262.1709794032.6958647.submit parsl.executors.status_handling INFO: Allocated block ID 6263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6263 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6263 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6263.1709794032.6962993.submit parsl.executors.status_handling INFO: Allocated block ID 6264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6264 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6264 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6264.1709794032.6967158.submit parsl.executors.status_handling INFO: Allocated block ID 6265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6265 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6265 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6265.1709794032.69717.submit parsl.executors.status_handling INFO: Allocated block ID 6266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6266 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6266 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6266.1709794032.6975975.submit parsl.executors.status_handling INFO: Allocated block ID 6267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6267 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6267 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6267.1709794032.6980503.submit parsl.executors.status_handling INFO: Allocated block ID 6268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6268 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6268 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6268.1709794032.6984925.submit parsl.executors.status_handling INFO: Allocated block ID 6269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6269 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6269 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6269.1709794032.6989405.submit parsl.executors.status_handling INFO: Allocated block ID 6270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6270 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6270 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6270.1709794032.6993701.submit parsl.executors.status_handling INFO: Allocated block ID 6271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6271 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6271 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6271.1709794032.6998034.submit parsl.executors.status_handling INFO: Allocated block ID 6272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6272 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6272 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6272.1709794032.700258.submit parsl.executors.status_handling INFO: Allocated block ID 6273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6273 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6273 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6273.1709794032.7006826.submit parsl.executors.status_handling INFO: Allocated block ID 6274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6274 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6274 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6274.1709794032.7011147.submit parsl.executors.status_handling INFO: Allocated block ID 6275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6275 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6275 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6275.1709794032.701542.submit parsl.executors.status_handling INFO: Allocated block ID 6276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6276 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6276 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6276.1709794032.7019796.submit parsl.executors.status_handling INFO: Allocated block ID 6277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6277 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6277 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6277.1709794032.7024148.submit parsl.executors.status_handling INFO: Allocated block ID 6278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6278 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6278 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6278.1709794032.7028553.submit parsl.executors.status_handling INFO: Allocated block ID 6279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6279 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6279 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6279.1709794032.7032943.submit parsl.executors.status_handling INFO: Allocated block ID 6280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6280 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6280 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6280.1709794032.7037222.submit parsl.executors.status_handling INFO: Allocated block ID 6281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6281 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6281 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6281.1709794032.7173028.submit parsl.executors.status_handling INFO: Allocated block ID 6282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6282 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6282 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6282.1709794032.7177534.submit parsl.executors.status_handling INFO: Allocated block ID 6283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6283 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6283 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6283.1709794032.7182033.submit parsl.executors.status_handling INFO: Allocated block ID 6284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6284 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6284 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6284.1709794032.718635.submit parsl.executors.status_handling INFO: Allocated block ID 6285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6285 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6285 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6285.1709794032.7190707.submit parsl.executors.status_handling INFO: Allocated block ID 6286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6286 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6286 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6286.1709794032.719499.submit parsl.executors.status_handling INFO: Allocated block ID 6287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6287 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6287 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6287.1709794032.7199364.submit parsl.executors.status_handling INFO: Allocated block ID 6288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6288 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6288 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6288.1709794032.7203624.submit parsl.executors.status_handling INFO: Allocated block ID 6289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6289 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6289 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6289.1709794032.72079.submit parsl.executors.status_handling INFO: Allocated block ID 6290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6290 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6290 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6290.1709794032.721229.submit parsl.executors.status_handling INFO: Allocated block ID 6291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6291 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6291 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6291.1709794032.7216585.submit parsl.executors.status_handling INFO: Allocated block ID 6292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6292 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6292 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6292.1709794032.7220907.submit parsl.executors.status_handling INFO: Allocated block ID 6293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6293 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6293 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6293.1709794032.7225168.submit parsl.executors.status_handling INFO: Allocated block ID 6294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6294 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6294 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6294.1709794032.7229414.submit parsl.executors.status_handling INFO: Allocated block ID 6295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6295 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6295 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6295.1709794032.7233715.submit parsl.executors.status_handling INFO: Allocated block ID 6296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6296 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6296 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6296.1709794032.7237988.submit parsl.executors.status_handling INFO: Allocated block ID 6297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6297 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6297 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6297.1709794032.7242372.submit parsl.executors.status_handling INFO: Allocated block ID 6298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6298 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6298 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6298.1709794032.7246652.submit parsl.executors.status_handling INFO: Allocated block ID 6299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6299 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6299 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6299.1709794032.725096.submit parsl.executors.status_handling INFO: Allocated block ID 6300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6300 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6300 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6300.1709794032.725514.submit parsl.executors.status_handling INFO: Allocated block ID 6301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6301 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6301 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6301.1709794032.7259443.submit parsl.executors.status_handling INFO: Allocated block ID 6302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6302 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6302 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6302.1709794032.7263749.submit parsl.executors.status_handling INFO: Allocated block ID 6303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6303 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6303 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6303.1709794032.726804.submit parsl.executors.status_handling INFO: Allocated block ID 6304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6304 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6304 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6304.1709794032.7272332.submit parsl.executors.status_handling INFO: Allocated block ID 6305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6305 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6305 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6305.1709794032.72766.submit parsl.executors.status_handling INFO: Allocated block ID 6306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6306 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6306 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6306.1709794032.728095.submit parsl.executors.status_handling INFO: Allocated block ID 6307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6307 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6307 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6307.1709794032.7285435.submit parsl.executors.status_handling INFO: Allocated block ID 6308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6308 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6308 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6308.1709794032.728982.submit parsl.executors.status_handling INFO: Allocated block ID 6309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6309 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6309 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6309.1709794032.7294166.submit parsl.executors.status_handling INFO: Allocated block ID 6310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6310 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6310 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6310.1709794032.7429287.submit parsl.executors.status_handling INFO: Allocated block ID 6311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6311 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6311 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6311.1709794032.743429.submit parsl.executors.status_handling INFO: Allocated block ID 6312 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6312 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6312 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6312 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6312 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6312.1709794032.7438846.submit parsl.executors.status_handling INFO: Allocated block ID 6313 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6313 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6313 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6313 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6313 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6313.1709794032.7443225.submit parsl.executors.status_handling INFO: Allocated block ID 6314 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6314 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6314 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6314 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6314 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6314.1709794032.7447507.submit parsl.executors.status_handling INFO: Allocated block ID 6315 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6315 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6315 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6315 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6315 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6315.1709794032.7452004.submit parsl.executors.status_handling INFO: Allocated block ID 6316 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6316 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6316 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6316 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6316 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6316.1709794032.7456408.submit parsl.executors.status_handling INFO: Allocated block ID 6317 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6317 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6317 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6317 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6317 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6317.1709794032.7460785.submit parsl.executors.status_handling INFO: Allocated block ID 6318 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6318 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6318 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6318 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6318 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6318.1709794032.7465115.submit parsl.executors.status_handling INFO: Allocated block ID 6319 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6319 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6319 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6319 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6319 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6319.1709794032.746955.submit parsl.executors.status_handling INFO: Allocated block ID 6320 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6320 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6320 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6320 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6320 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6320.1709794032.7473814.submit parsl.executors.status_handling INFO: Allocated block ID 6321 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6321 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6321 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6321 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6321 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6321.1709794032.7478173.submit parsl.executors.status_handling INFO: Allocated block ID 6322 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6322 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6322 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6322 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6322 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6322.1709794032.748251.submit parsl.executors.status_handling INFO: Allocated block ID 6323 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6323 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6323 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6323 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6323 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6323.1709794032.74868.submit parsl.executors.status_handling INFO: Allocated block ID 6324 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6324 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6324 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6324 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6324 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6324.1709794032.749107.submit parsl.executors.status_handling INFO: Allocated block ID 6325 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6325 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6325 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6325 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6325 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6325.1709794032.749533.submit parsl.executors.status_handling INFO: Allocated block ID 6326 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6326 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6326 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6326 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6326 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6326.1709794032.749962.submit parsl.executors.status_handling INFO: Allocated block ID 6327 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6327 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6327 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6327 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6327 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6327.1709794032.7503974.submit parsl.executors.status_handling INFO: Allocated block ID 6328 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6328 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6328 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6328 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6328 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6328.1709794032.7508278.submit parsl.executors.status_handling INFO: Allocated block ID 6329 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6329 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6329 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6329 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6329 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6329.1709794032.7512538.submit parsl.executors.status_handling INFO: Allocated block ID 6330 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6330 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6330 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6330 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6330 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6330.1709794032.7516794.submit parsl.executors.status_handling INFO: Allocated block ID 6331 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6331 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6331 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6331 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6331 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6331.1709794032.7521105.submit parsl.executors.status_handling INFO: Allocated block ID 6332 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6332 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6332 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6332 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6332 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6332.1709794032.7525344.submit parsl.executors.status_handling INFO: Allocated block ID 6333 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6333 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6333 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6333 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6333 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6333.1709794032.7529626.submit parsl.executors.status_handling INFO: Allocated block ID 6334 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6334 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6334 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6334 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6334 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6334.1709794032.7533898.submit parsl.executors.status_handling INFO: Allocated block ID 6335 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6335 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6335 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6335 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6335 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6335.1709794032.753815.submit parsl.executors.status_handling INFO: Allocated block ID 6336 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6336 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6336 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6336 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6336 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6336.1709794032.7542338.submit parsl.executors.status_handling INFO: Allocated block ID 6337 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6337 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6337 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6337 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6337 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6337.1709794032.7546577.submit parsl.executors.status_handling INFO: Allocated block ID 6338 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6338 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6338 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6338 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6338 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6338.1709794032.7550876.submit parsl.executors.status_handling INFO: Allocated block ID 6339 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6339 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6339 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6339 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6339 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6339.1709794032.7555182.submit parsl.executors.status_handling INFO: Allocated block ID 6340 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6340 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6340 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6340 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6340 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6340.1709794032.7690408.submit parsl.executors.status_handling INFO: Allocated block ID 6341 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6341 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6341 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6341 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6341 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6341.1709794032.7695167.submit parsl.executors.status_handling INFO: Allocated block ID 6342 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6342 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6342 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6342 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6342 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6342.1709794032.769959.submit parsl.executors.status_handling INFO: Allocated block ID 6343 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6343 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6343 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6343 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6343 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6343.1709794032.770394.submit parsl.executors.status_handling INFO: Allocated block ID 6344 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6344 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6344 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6344 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6344 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6344.1709794032.7708323.submit parsl.executors.status_handling INFO: Allocated block ID 6345 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6345 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6345 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6345 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6345 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6345.1709794032.7712572.submit parsl.executors.status_handling INFO: Allocated block ID 6346 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6346 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6346 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6346 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6346 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6346.1709794032.7716813.submit parsl.executors.status_handling INFO: Allocated block ID 6347 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6347 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6347 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6347 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6347 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6347.1709794032.772122.submit parsl.executors.status_handling INFO: Allocated block ID 6348 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6348 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6348 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6348 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6348 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6348.1709794032.7725475.submit parsl.executors.status_handling INFO: Allocated block ID 6349 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6349 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6349 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6349 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6349 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6349.1709794032.7729814.submit parsl.executors.status_handling INFO: Allocated block ID 6350 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6350 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6350 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6350 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6350 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6350.1709794032.7734048.submit parsl.executors.status_handling INFO: Allocated block ID 6351 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6351 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6351 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6351 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6351 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6351.1709794032.7738414.submit parsl.executors.status_handling INFO: Allocated block ID 6352 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6352 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6352 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6352 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6352 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6352.1709794032.7742689.submit parsl.executors.status_handling INFO: Allocated block ID 6353 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6353 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6353 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6353 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6353 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6353.1709794032.7746925.submit parsl.executors.status_handling INFO: Allocated block ID 6354 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6354 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6354 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6354 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6354 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6354.1709794032.775123.submit parsl.executors.status_handling INFO: Allocated block ID 6355 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6355 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6355 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6355 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6355 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6355.1709794032.7755523.submit parsl.executors.status_handling INFO: Allocated block ID 6356 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6356 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6356 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6356 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6356 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6356.1709794032.7759793.submit parsl.executors.status_handling INFO: Allocated block ID 6357 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6357 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6357 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6357 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6357 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6357.1709794032.7764063.submit parsl.executors.status_handling INFO: Allocated block ID 6358 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6358 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6358 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6358 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6358 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6358.1709794032.776837.submit parsl.executors.status_handling INFO: Allocated block ID 6359 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6359 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6359 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6359 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6359 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6359.1709794032.7772596.submit parsl.executors.status_handling INFO: Allocated block ID 6360 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6360 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6360 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6360 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6360 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6360.1709794032.7776797.submit parsl.executors.status_handling INFO: Allocated block ID 6361 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6361 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6361 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6361 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6361 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6361.1709794032.778105.submit parsl.executors.status_handling INFO: Allocated block ID 6362 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6362 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6362 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6362 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6362 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6362.1709794032.778524.submit parsl.executors.status_handling INFO: Allocated block ID 6363 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6363 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6363 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6363 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6363 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6363.1709794032.7789605.submit parsl.executors.status_handling INFO: Allocated block ID 6364 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6364 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6364 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6364 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6364 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6364.1709794032.7793849.submit parsl.executors.status_handling INFO: Allocated block ID 6365 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6365 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6365 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6365 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6365 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6365.1709794032.7798119.submit parsl.executors.status_handling INFO: Allocated block ID 6366 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6366 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6366 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6366 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6366 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6366.1709794032.780244.submit parsl.executors.status_handling INFO: Allocated block ID 6367 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6367 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6367 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6367 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6367 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6367.1709794032.7806687.submit parsl.executors.status_handling INFO: Allocated block ID 6368 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6368 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6368 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6368 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6368 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6368.1709794032.7810955.submit parsl.executors.status_handling INFO: Allocated block ID 6369 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6369 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6369 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6369 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6369 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6369.1709794032.781519.submit parsl.executors.status_handling INFO: Allocated block ID 6370 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6370 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6370 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6370 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6370 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6370.1709794032.7950406.submit parsl.executors.status_handling INFO: Allocated block ID 6371 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6371 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6371 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6371 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6371 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6371.1709794032.7955093.submit parsl.executors.status_handling INFO: Allocated block ID 6372 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6372 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6372 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6372 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6372 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6372.1709794032.7959473.submit parsl.executors.status_handling INFO: Allocated block ID 6373 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6373 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6373 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6373 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6373 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6373.1709794032.7963772.submit parsl.executors.status_handling INFO: Allocated block ID 6374 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6374 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6374 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6374 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6374 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6374.1709794032.7967985.submit parsl.executors.status_handling INFO: Allocated block ID 6375 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6375 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6375 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6375 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6375 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6375.1709794032.7972379.submit parsl.executors.status_handling INFO: Allocated block ID 6376 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6376 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6376 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6376 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6376 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6376.1709794032.7976654.submit parsl.executors.status_handling INFO: Allocated block ID 6377 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6377 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6377 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6377 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6377 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6377.1709794032.7980964.submit parsl.executors.status_handling INFO: Allocated block ID 6378 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6378 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6378 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6378 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6378 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6378.1709794032.7985196.submit parsl.executors.status_handling INFO: Allocated block ID 6379 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6379 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6379 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6379 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6379 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6379.1709794032.7989488.submit parsl.executors.status_handling INFO: Allocated block ID 6380 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6380 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6380 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6380 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6380 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6380.1709794032.7993774.submit parsl.executors.status_handling INFO: Allocated block ID 6381 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6381 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6381 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6381 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6381 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6381.1709794032.799797.submit parsl.executors.status_handling INFO: Allocated block ID 6382 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6382 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6382 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6382 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6382 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6382.1709794032.8002226.submit parsl.executors.status_handling INFO: Allocated block ID 6383 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6383 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6383 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6383 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6383 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6383.1709794032.8006482.submit parsl.executors.status_handling INFO: Allocated block ID 6384 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6384 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6384 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6384 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6384 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6384.1709794032.80108.submit parsl.executors.status_handling INFO: Allocated block ID 6385 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6385 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6385 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6385 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6385 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6385.1709794032.8015003.submit parsl.executors.status_handling INFO: Allocated block ID 6386 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6386 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6386 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6386 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6386 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6386.1709794032.8019238.submit parsl.executors.status_handling INFO: Allocated block ID 6387 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6387 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6387 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6387 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6387 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6387.1709794032.8023615.submit parsl.executors.status_handling INFO: Allocated block ID 6388 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6388 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6388 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6388 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6388 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6388.1709794032.802785.submit parsl.executors.status_handling INFO: Allocated block ID 6389 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6389 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6389 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6389 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6389 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6389.1709794032.8032205.submit parsl.executors.status_handling INFO: Allocated block ID 6390 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6390 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6390 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6390 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6390 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6390.1709794032.8036475.submit parsl.executors.status_handling INFO: Allocated block ID 6391 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6391 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6391 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6391 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6391 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6391.1709794032.8040805.submit parsl.executors.status_handling INFO: Allocated block ID 6392 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6392 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6392 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6392 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6392 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6392.1709794032.8045115.submit parsl.executors.status_handling INFO: Allocated block ID 6393 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6393 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6393 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6393 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6393 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6393.1709794032.8049386.submit parsl.executors.status_handling INFO: Allocated block ID 6394 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6394 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6394 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6394 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6394 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6394.1709794032.805362.submit parsl.executors.status_handling INFO: Allocated block ID 6395 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6395 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6395 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6395 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6395 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6395.1709794032.805787.submit parsl.executors.status_handling INFO: Allocated block ID 6396 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6396 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6396 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6396 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6396 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6396.1709794032.8062096.submit parsl.executors.status_handling INFO: Allocated block ID 6397 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6397 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6397 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6397 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6397 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6397.1709794032.8066278.submit parsl.executors.status_handling INFO: Allocated block ID 6398 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6398 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6398 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6398 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6398 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6398.1709794032.8070498.submit parsl.executors.status_handling INFO: Allocated block ID 6399 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6399 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6399 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6399 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6399 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6399.1709794032.8074732.submit parsl.executors.status_handling INFO: Allocated block ID 6400 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6400 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6400 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6400 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6400 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6400.1709794032.8209977.submit parsl.executors.status_handling INFO: Allocated block ID 6401 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6401 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6401 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6401 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6401 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6401.1709794032.8214872.submit parsl.executors.status_handling INFO: Allocated block ID 6402 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6402 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6402 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6402 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6402 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6402.1709794032.8219395.submit parsl.executors.status_handling INFO: Allocated block ID 6403 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6403 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6403 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6403 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6403 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6403.1709794032.8223925.submit parsl.executors.status_handling INFO: Allocated block ID 6404 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6404 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6404 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6404 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6404 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6404.1709794032.8228388.submit parsl.executors.status_handling INFO: Allocated block ID 6405 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6405 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6405 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6405 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6405 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6405.1709794032.8232722.submit parsl.executors.status_handling INFO: Allocated block ID 6406 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6406 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6406 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6406 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6406 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6406.1709794032.823706.submit parsl.executors.status_handling INFO: Allocated block ID 6407 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6407 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6407 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6407 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6407 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6407.1709794032.824147.submit parsl.executors.status_handling INFO: Allocated block ID 6408 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6408 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6408 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6408 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6408 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6408.1709794032.8245869.submit parsl.executors.status_handling INFO: Allocated block ID 6409 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6409 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6409 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6409 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6409 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6409.1709794032.8250332.submit parsl.executors.status_handling INFO: Allocated block ID 6410 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6410 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6410 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6410 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6410 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6410.1709794032.8254664.submit parsl.executors.status_handling INFO: Allocated block ID 6411 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6411 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6411 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6411 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6411 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6411.1709794032.825916.submit parsl.executors.status_handling INFO: Allocated block ID 6412 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6412 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6412 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6412 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6412 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6412.1709794032.8263593.submit parsl.executors.status_handling INFO: Allocated block ID 6413 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6413 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6413 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6413 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6413 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6413.1709794032.826799.submit parsl.executors.status_handling INFO: Allocated block ID 6414 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6414 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6414 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6414 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6414 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6414.1709794032.8272452.submit parsl.executors.status_handling INFO: Allocated block ID 6415 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6415 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6415 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6415 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6415 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6415.1709794032.8276706.submit parsl.executors.status_handling INFO: Allocated block ID 6416 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6416 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6416 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6416 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6416 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6416.1709794032.8280985.submit parsl.executors.status_handling INFO: Allocated block ID 6417 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6417 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6417 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6417 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6417 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6417.1709794032.8285284.submit parsl.executors.status_handling INFO: Allocated block ID 6418 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6418 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6418 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6418 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6418 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6418.1709794032.82896.submit parsl.executors.status_handling INFO: Allocated block ID 6419 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6419 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6419 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6419 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6419 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6419.1709794032.82938.submit parsl.executors.status_handling INFO: Allocated block ID 6420 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6420 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6420 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6420 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6420 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6420.1709794032.8297966.submit parsl.executors.status_handling INFO: Allocated block ID 6421 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6421 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6421 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6421 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6421 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6421.1709794032.8302286.submit parsl.executors.status_handling INFO: Allocated block ID 6422 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6422 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6422 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6422 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6422 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6422.1709794032.830651.submit parsl.executors.status_handling INFO: Allocated block ID 6423 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6423 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6423 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6423 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6423 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6423.1709794032.8310857.submit parsl.executors.status_handling INFO: Allocated block ID 6424 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6424 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6424 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6424 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6424 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6424.1709794032.8315096.submit parsl.executors.status_handling INFO: Allocated block ID 6425 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6425 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6425 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6425 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6425 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6425.1709794032.8319376.submit parsl.executors.status_handling INFO: Allocated block ID 6426 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6426 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6426 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6426 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6426 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6426.1709794032.8323612.submit parsl.executors.status_handling INFO: Allocated block ID 6427 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6427 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6427 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6427 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6427 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6427.1709794032.832785.submit parsl.executors.status_handling INFO: Allocated block ID 6428 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6428 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6428 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6428 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6428 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6428.1709794032.8332155.submit parsl.executors.status_handling INFO: Allocated block ID 6429 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6429 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6429 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6429 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6429 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6429.1709794032.8336413.submit parsl.executors.status_handling INFO: Allocated block ID 6430 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6430 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6430 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6430 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6430 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6430.1709794032.8472025.submit parsl.executors.status_handling INFO: Allocated block ID 6431 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6431 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6431 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6431 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6431 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6431.1709794032.847668.submit parsl.executors.status_handling INFO: Allocated block ID 6432 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6432 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6432 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6432 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6432 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6432.1709794032.848116.submit parsl.executors.status_handling INFO: Allocated block ID 6433 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6433 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6433 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6433 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6433 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6433.1709794032.848548.submit parsl.executors.status_handling INFO: Allocated block ID 6434 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6434 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6434 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6434 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6434 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6434.1709794032.8489788.submit parsl.executors.status_handling INFO: Allocated block ID 6435 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6435 2024-03-06 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6435 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6435 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6435 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6435.1709794032.8494065.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 464a683a61e426528b8d8e1dcfc4b018 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a27308c-a666-4e61-b670-b3e603d61632_biasIsr_1123736_8.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123736/6a27308c-a666-4e61-b670-b3e603d61632_biasIsr_1123736_8.stderr 2024-03-06 22:47:17 proc_lsst.multi:146 [INFO] found job 76135 in provider local proc_lsst.multi INFO: found job 76135 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1331 parsl.dataflow.strategy DEBUG: Executor multi has 1331 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 6436 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6436 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6436 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6436 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6436 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6436.1709794037.8130991.submit parsl.executors.status_handling INFO: Allocated block ID 6437 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6437 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6437 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6437 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6437 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6437.1709794037.813654.submit parsl.executors.status_handling INFO: Allocated block ID 6438 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6438 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6438 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6438 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6438 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6438.1709794037.8141322.submit parsl.executors.status_handling INFO: Allocated block ID 6439 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6439 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6439 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6439 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6439 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6439.1709794037.814582.submit parsl.executors.status_handling INFO: Allocated block ID 6440 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6440 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6440 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6440 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6440 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6440.1709794037.815023.submit parsl.executors.status_handling INFO: Allocated block ID 6441 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6441 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6441 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6441 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6441 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6441.1709794037.8154573.submit parsl.executors.status_handling INFO: Allocated block ID 6442 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6442 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6442 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6442 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6442 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6442.1709794037.8159935.submit parsl.executors.status_handling INFO: Allocated block ID 6443 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6443 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6443 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6443 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6443 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6443.1709794037.8164306.submit parsl.executors.status_handling INFO: Allocated block ID 6444 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6444 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6444 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6444 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6444 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6444.1709794037.8168597.submit parsl.executors.status_handling INFO: Allocated block ID 6445 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6445 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6445 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6445 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6445 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6445.1709794037.8172815.submit parsl.executors.status_handling INFO: Allocated block ID 6446 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6446 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6446 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6446 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6446 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6446.1709794037.817699.submit parsl.executors.status_handling INFO: Allocated block ID 6447 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6447 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6447 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6447 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6447 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6447.1709794037.8181305.submit parsl.executors.status_handling INFO: Allocated block ID 6448 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6448 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6448 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6448 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6448 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6448.1709794037.818553.submit parsl.executors.status_handling INFO: Allocated block ID 6449 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6449 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6449 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6449 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6449 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6449.1709794037.818986.submit parsl.executors.status_handling INFO: Allocated block ID 6450 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6450 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6450 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6450 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6450 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6450.1709794037.819411.submit parsl.executors.status_handling INFO: Allocated block ID 6451 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6451 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6451 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6451 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6451 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6451.1709794037.8198388.submit parsl.executors.status_handling INFO: Allocated block ID 6452 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6452 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6452 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6452 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6452 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6452.1709794037.8202631.submit parsl.executors.status_handling INFO: Allocated block ID 6453 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6453 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6453 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6453 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6453 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6453.1709794037.820686.submit parsl.executors.status_handling INFO: Allocated block ID 6454 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6454 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6454 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6454 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6454 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6454.1709794037.8211167.submit parsl.executors.status_handling INFO: Allocated block ID 6455 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6455 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6455 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6455 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6455 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6455.1709794037.8215384.submit parsl.executors.status_handling INFO: Allocated block ID 6456 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6456 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6456 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6456 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6456 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6456.1709794037.8219638.submit parsl.executors.status_handling INFO: Allocated block ID 6457 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6457 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6457 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6457 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6457 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6457.1709794037.822384.submit parsl.executors.status_handling INFO: Allocated block ID 6458 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6458 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6458 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6458 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6458 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6458.1709794037.8228014.submit parsl.executors.status_handling INFO: Allocated block ID 6459 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6459 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6459 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6459 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6459 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6459.1709794037.8232296.submit parsl.executors.status_handling INFO: Allocated block ID 6460 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6460 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6460 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6460 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6460 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6460.1709794037.8236487.submit parsl.executors.status_handling INFO: Allocated block ID 6461 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6461 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6461 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6461 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6461 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6461.1709794037.824083.submit parsl.executors.status_handling INFO: Allocated block ID 6462 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6462 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6462 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6462 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6462 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6462.1709794037.8245072.submit parsl.executors.status_handling INFO: Allocated block ID 6463 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6463 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6463 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6463 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6463 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6463.1709794037.8249314.submit parsl.executors.status_handling INFO: Allocated block ID 6464 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6464 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6464 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6464 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6464 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6464.1709794037.8253562.submit parsl.executors.status_handling INFO: Allocated block ID 6465 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6465 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6465 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6465 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6465 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6465.1709794037.825777.submit parsl.executors.status_handling INFO: Allocated block ID 6466 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6466 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6466 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6466 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6466 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6466.1709794037.8262126.submit parsl.executors.status_handling INFO: Allocated block ID 6467 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6467 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6467 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6467 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6467 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6467.1709794037.826632.submit parsl.executors.status_handling INFO: Allocated block ID 6468 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6468 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6468 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6468 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6468 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6468.1709794037.8477116.submit parsl.executors.status_handling INFO: Allocated block ID 6469 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6469 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6469 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6469 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6469 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6469.1709794037.8482058.submit parsl.executors.status_handling INFO: Allocated block ID 6470 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6470 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6470 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6470 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6470 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6470.1709794037.8486338.submit parsl.executors.status_handling INFO: Allocated block ID 6471 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6471 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6471 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6471 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6471 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6471.1709794037.8490686.submit parsl.executors.status_handling INFO: Allocated block ID 6472 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6472 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6472 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6472 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6472 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6472.1709794037.8495092.submit parsl.executors.status_handling INFO: Allocated block ID 6473 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6473 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6473 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6473 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6473 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6473.1709794037.8499472.submit parsl.executors.status_handling INFO: Allocated block ID 6474 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6474 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6474 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6474 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6474 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6474.1709794037.8503742.submit parsl.executors.status_handling INFO: Allocated block ID 6475 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6475 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6475 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6475 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6475 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6475.1709794037.8507984.submit parsl.executors.status_handling INFO: Allocated block ID 6476 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6476 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6476 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6476 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6476 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6476.1709794037.851224.submit parsl.executors.status_handling INFO: Allocated block ID 6477 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6477 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6477 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6477 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6477 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6477.1709794037.8516486.submit parsl.executors.status_handling INFO: Allocated block ID 6478 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6478 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6478 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6478 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6478 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6478.1709794037.8520787.submit parsl.executors.status_handling INFO: Allocated block ID 6479 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6479 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6479 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6479 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6479 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6479.1709794037.8524992.submit parsl.executors.status_handling INFO: Allocated block ID 6480 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6480 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6480 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6480 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6480 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6480.1709794037.8529406.submit parsl.executors.status_handling INFO: Allocated block ID 6481 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6481 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6481 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6481 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6481 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6481.1709794037.853366.submit parsl.executors.status_handling INFO: Allocated block ID 6482 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6482 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6482 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6482 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6482 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6482.1709794037.8537886.submit parsl.executors.status_handling INFO: Allocated block ID 6483 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6483 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6483 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6483 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6483 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6483.1709794037.854217.submit parsl.executors.status_handling INFO: Allocated block ID 6484 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6484 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6484 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6484 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6484 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6484.1709794037.8546405.submit parsl.executors.status_handling INFO: Allocated block ID 6485 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6485 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6485 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6485 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6485 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6485.1709794037.855079.submit parsl.executors.status_handling INFO: Allocated block ID 6486 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6486 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6486 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6486 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6486 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6486.1709794037.8555028.submit parsl.executors.status_handling INFO: Allocated block ID 6487 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6487 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6487 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6487 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6487 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6487.1709794037.8559294.submit parsl.executors.status_handling INFO: Allocated block ID 6488 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6488 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6488 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6488 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6488 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6488.1709794037.8563533.submit parsl.executors.status_handling INFO: Allocated block ID 6489 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6489 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6489 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6489 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6489 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6489.1709794037.856776.submit parsl.executors.status_handling INFO: Allocated block ID 6490 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6490 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6490 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6490 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6490 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6490.1709794037.8572054.submit parsl.executors.status_handling INFO: Allocated block ID 6491 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6491 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6491 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6491 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6491 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6491.1709794037.857634.submit parsl.executors.status_handling INFO: Allocated block ID 6492 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6492 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6492 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6492 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6492 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6492.1709794037.8685532.submit parsl.executors.status_handling INFO: Allocated block ID 6493 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6493 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6493 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6493 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6493 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6493.1709794037.8690464.submit parsl.executors.status_handling INFO: Allocated block ID 6494 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6494 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6494 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6494 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6494 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6494.1709794037.8694828.submit parsl.executors.status_handling INFO: Allocated block ID 6495 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6495 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6495 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6495 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6495 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6495.1709794037.869919.submit parsl.executors.status_handling INFO: Allocated block ID 6496 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6496 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6496 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6496 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6496 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6496.1709794037.8703492.submit parsl.executors.status_handling INFO: Allocated block ID 6497 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6497 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6497 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6497 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6497 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6497.1709794037.8707838.submit parsl.executors.status_handling INFO: Allocated block ID 6498 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6498 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6498 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6498 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6498 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6498.1709794037.8712323.submit parsl.executors.status_handling INFO: Allocated block ID 6499 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6499 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6499 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6499 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6499 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6499.1709794037.871662.submit parsl.executors.status_handling INFO: Allocated block ID 6500 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6500 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6500 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6500 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6500 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6500.1709794037.8721082.submit parsl.executors.status_handling INFO: Allocated block ID 6501 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6501 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6501 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6501 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6501 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6501.1709794037.8725436.submit parsl.executors.status_handling INFO: Allocated block ID 6502 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6502 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6502 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6502 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6502 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6502.1709794037.8729818.submit parsl.executors.status_handling INFO: Allocated block ID 6503 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6503 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6503 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6503 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6503 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6503.1709794037.8734057.submit parsl.executors.status_handling INFO: Allocated block ID 6504 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6504 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6504 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6504 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6504 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6504.1709794037.8738399.submit parsl.executors.status_handling INFO: Allocated block ID 6505 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6505 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6505 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6505 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6505 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6505.1709794037.8742723.submit parsl.executors.status_handling INFO: Allocated block ID 6506 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6506 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6506 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6506 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6506 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6506.1709794037.874705.submit parsl.executors.status_handling INFO: Allocated block ID 6507 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6507 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6507 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6507 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6507 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6507.1709794037.8751562.submit parsl.executors.status_handling INFO: Allocated block ID 6508 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6508 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6508 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6508 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6508 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6508.1709794037.8755872.submit parsl.executors.status_handling INFO: Allocated block ID 6509 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6509 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6509 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6509 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6509 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6509.1709794037.87603.submit parsl.executors.status_handling INFO: Allocated block ID 6510 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6510 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6510 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6510 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6510 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6510.1709794037.8764591.submit parsl.executors.status_handling INFO: Allocated block ID 6511 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6511 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6511 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6511 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6511 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6511.1709794037.8768883.submit parsl.executors.status_handling INFO: Allocated block ID 6512 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6512 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6512 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6512 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6512 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6512.1709794037.8773334.submit parsl.executors.status_handling INFO: Allocated block ID 6513 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6513 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6513 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6513 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6513 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6513.1709794037.877768.submit parsl.executors.status_handling INFO: Allocated block ID 6514 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6514 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6514 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6514 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6514 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6514.1709794037.878221.submit parsl.executors.status_handling INFO: Allocated block ID 6515 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6515 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6515 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6515 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6515 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6515.1709794037.87867.submit parsl.executors.status_handling INFO: Allocated block ID 6516 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6516 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6516 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6516 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6516 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6516.1709794037.899881.submit parsl.executors.status_handling INFO: Allocated block ID 6517 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6517 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6517 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6517 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6517 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6517.1709794037.9003856.submit parsl.executors.status_handling INFO: Allocated block ID 6518 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6518 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6518 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6518 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6518 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6518.1709794037.900843.submit parsl.executors.status_handling INFO: Allocated block ID 6519 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6519 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6519 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6519 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6519 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6519.1709794037.901277.submit parsl.executors.status_handling INFO: Allocated block ID 6520 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6520 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6520 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6520 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6520 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6520.1709794037.9017122.submit parsl.executors.status_handling INFO: Allocated block ID 6521 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6521 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6521 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6521 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6521 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6521.1709794037.9021564.submit parsl.executors.status_handling INFO: Allocated block ID 6522 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6522 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6522 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6522 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6522 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6522.1709794037.9025924.submit parsl.executors.status_handling INFO: Allocated block ID 6523 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6523 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6523 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6523 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6523 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6523.1709794037.9030309.submit parsl.executors.status_handling INFO: Allocated block ID 6524 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6524 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6524 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6524 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6524 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6524.1709794037.903464.submit parsl.executors.status_handling INFO: Allocated block ID 6525 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6525 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6525 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6525 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6525 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6525.1709794037.9039085.submit parsl.executors.status_handling INFO: Allocated block ID 6526 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6526 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6526 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6526 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6526 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6526.1709794037.9043424.submit parsl.executors.status_handling INFO: Allocated block ID 6527 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6527 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6527 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6527 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6527 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6527.1709794037.9047759.submit parsl.executors.status_handling INFO: Allocated block ID 6528 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6528 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6528 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6528 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6528 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6528.1709794037.905219.submit parsl.executors.status_handling INFO: Allocated block ID 6529 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6529 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6529 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6529 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6529 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6529.1709794037.9056396.submit parsl.executors.status_handling INFO: Allocated block ID 6530 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6530 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6530 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6530 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6530 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6530.1709794037.906068.submit parsl.executors.status_handling INFO: Allocated block ID 6531 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6531 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6531 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6531 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6531 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6531.1709794037.9064882.submit parsl.executors.status_handling INFO: Allocated block ID 6532 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6532 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6532 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6532 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6532 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6532.1709794037.906918.submit parsl.executors.status_handling INFO: Allocated block ID 6533 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6533 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6533 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6533 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6533 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6533.1709794037.907344.submit parsl.executors.status_handling INFO: Allocated block ID 6534 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6534 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6534 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6534 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6534 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6534.1709794037.9077635.submit parsl.executors.status_handling INFO: Allocated block ID 6535 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6535 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6535 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6535 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6535 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6535.1709794037.9181893.submit parsl.executors.status_handling INFO: Allocated block ID 6536 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6536 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6536 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6536 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6536 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6536.1709794037.9187558.submit parsl.executors.status_handling INFO: Allocated block ID 6537 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6537 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6537 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6537 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6537 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6537.1709794037.9191926.submit parsl.executors.status_handling INFO: Allocated block ID 6538 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6538 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6538 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6538 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6538.1709794037.9196205.submit parsl.executors.status_handling INFO: Allocated block ID 6539 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6539 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6539 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6539 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6539 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6539.1709794037.9200552.submit parsl.executors.status_handling INFO: Allocated block ID 6540 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6540 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6540 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6540 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6540 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6540.1709794037.9204767.submit parsl.executors.status_handling INFO: Allocated block ID 6541 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6541 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6541 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6541 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6541 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6541.1709794037.9209049.submit parsl.executors.status_handling INFO: Allocated block ID 6542 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6542 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6542 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6542 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6542 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6542.1709794037.9213233.submit parsl.executors.status_handling INFO: Allocated block ID 6543 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6543 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6543 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6543 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6543 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6543.1709794037.921741.submit parsl.executors.status_handling INFO: Allocated block ID 6544 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6544 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6544 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6544 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6544 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6544.1709794037.9221702.submit parsl.executors.status_handling INFO: Allocated block ID 6545 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6545 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6545 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6545 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6545 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6545.1709794037.9225967.submit parsl.executors.status_handling INFO: Allocated block ID 6546 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6546 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6546 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6546 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6546 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6546.1709794037.9230218.submit parsl.executors.status_handling INFO: Allocated block ID 6547 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6547 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6547 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6547 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6547 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6547.1709794037.9234376.submit parsl.executors.status_handling INFO: Allocated block ID 6548 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6548 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6548 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6548 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6548 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6548.1709794037.9238555.submit parsl.executors.status_handling INFO: Allocated block ID 6549 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6549 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6549 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6549 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6549 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6549.1709794037.9242709.submit parsl.executors.status_handling INFO: Allocated block ID 6550 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6550 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6550 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6550 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6550 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6550.1709794037.924685.submit parsl.executors.status_handling INFO: Allocated block ID 6551 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6551 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6551 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6551 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6551 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6551.1709794037.925105.submit parsl.executors.status_handling INFO: Allocated block ID 6552 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6552 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6552 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6552 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6552 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6552.1709794037.9255173.submit parsl.executors.status_handling INFO: Allocated block ID 6553 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6553 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6553 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6553 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6553 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6553.1709794037.9259388.submit parsl.executors.status_handling INFO: Allocated block ID 6554 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6554 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6554 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6554 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6554 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6554.1709794037.926351.submit parsl.executors.status_handling INFO: Allocated block ID 6555 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6555 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6555 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6555 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6555 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6555.1709794037.926759.submit parsl.executors.status_handling INFO: Allocated block ID 6556 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6556 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6556 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6556 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6556 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6556.1709794037.9271812.submit parsl.executors.status_handling INFO: Allocated block ID 6557 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6557 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6557 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6557 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6557 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6557.1709794037.9276106.submit parsl.executors.status_handling INFO: Allocated block ID 6558 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6558 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6558 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6558 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6558 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6558.1709794037.92803.submit parsl.executors.status_handling INFO: Allocated block ID 6559 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6559 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6559 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6559 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6559 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6559.1709794037.9284525.submit parsl.executors.status_handling INFO: Allocated block ID 6560 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6560 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6560 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6560 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6560 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6560.1709794037.9290862.submit parsl.executors.status_handling INFO: Allocated block ID 6561 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6561 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6561 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6561 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6561 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6561.1709794037.9295325.submit parsl.executors.status_handling INFO: Allocated block ID 6562 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6562 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6562 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6562 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6562 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6562.1709794037.929963.submit parsl.executors.status_handling INFO: Allocated block ID 6563 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6563 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6563 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6563 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6563 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6563.1709794037.9303796.submit parsl.executors.status_handling INFO: Allocated block ID 6564 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6564 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6564 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6564 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6564 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6564.1709794037.9307954.submit parsl.executors.status_handling INFO: Allocated block ID 6565 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6565 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6565 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6565 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6565 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6565.1709794037.9312174.submit parsl.executors.status_handling INFO: Allocated block ID 6566 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6566 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6566 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6566 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6566 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6566.1709794037.9316282.submit parsl.executors.status_handling INFO: Allocated block ID 6567 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6567 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6567 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6567 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6567 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6567.1709794037.9320495.submit parsl.executors.status_handling INFO: Allocated block ID 6568 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6568 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6568 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6568 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6568 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6568.1709794037.9324672.submit parsl.executors.status_handling INFO: Allocated block ID 6569 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6569 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6569 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6569 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6569 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6569.1709794037.9328942.submit parsl.executors.status_handling INFO: Allocated block ID 6570 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6570 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6570 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6570 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6570 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6570.1709794037.9333122.submit parsl.executors.status_handling INFO: Allocated block ID 6571 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6571 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6571 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6571 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6571 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6571.1709794037.9337263.submit parsl.executors.status_handling INFO: Allocated block ID 6572 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6572 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6572 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6572 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6572 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6572.1709794037.934148.submit parsl.executors.status_handling INFO: Allocated block ID 6573 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6573 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6573 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6573 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6573 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6573.1709794037.9345622.submit parsl.executors.status_handling INFO: Allocated block ID 6574 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6574 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6574 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6574 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6574 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6574.1709794037.9349868.submit parsl.executors.status_handling INFO: Allocated block ID 6575 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6575 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6575 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6575 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6575 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6575.1709794037.9354036.submit parsl.executors.status_handling INFO: Allocated block ID 6576 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6576 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6576 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6576 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6576 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6576.1709794037.9358208.submit parsl.executors.status_handling INFO: Allocated block ID 6577 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6577 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6577 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6577 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6577 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6577.1709794037.9362557.submit parsl.executors.status_handling INFO: Allocated block ID 6578 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6578 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6578 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6578 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6578 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6578.1709794037.936686.submit parsl.executors.status_handling INFO: Allocated block ID 6579 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6579 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6579 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6579 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6579 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6579.1709794037.9371223.submit parsl.executors.status_handling INFO: Allocated block ID 6580 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6580 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6580 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6580 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6580 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6580.1709794037.937551.submit parsl.executors.status_handling INFO: Allocated block ID 6581 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6581 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6581 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6581 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6581 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6581.1709794037.93799.submit parsl.executors.status_handling INFO: Allocated block ID 6582 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6582 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6582 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6582 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6582 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6582.1709794037.938421.submit parsl.executors.status_handling INFO: Allocated block ID 6583 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6583 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6583 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6583 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6583 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6583.1709794037.9388597.submit parsl.executors.status_handling INFO: Allocated block ID 6584 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6584 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6584 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6584 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6584 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6584.1709794037.9392977.submit parsl.executors.status_handling INFO: Allocated block ID 6585 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6585 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6585 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6585 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6585 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6585.1709794037.9397259.submit parsl.executors.status_handling INFO: Allocated block ID 6586 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6586 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6586 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6586 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6586 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6586.1709794037.9483268.submit parsl.executors.status_handling INFO: Allocated block ID 6587 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6587 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6587 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6587 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6587 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6587.1709794037.9487567.submit parsl.executors.status_handling INFO: Allocated block ID 6588 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6588 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6588 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6588 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6588 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6588.1709794037.9491806.submit parsl.executors.status_handling INFO: Allocated block ID 6589 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6589 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6589 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6589 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6589 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6589.1709794037.9495997.submit parsl.executors.status_handling INFO: Allocated block ID 6590 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6590 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6590 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6590 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6590 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6590.1709794037.9500306.submit parsl.executors.status_handling INFO: Allocated block ID 6591 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6591 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6591 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6591 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6591 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6591.1709794037.9504502.submit parsl.executors.status_handling INFO: Allocated block ID 6592 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6592 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6592 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6592 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6592 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6592.1709794037.9508727.submit parsl.executors.status_handling INFO: Allocated block ID 6593 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6593 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6593 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6593 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6593 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6593.1709794037.9512973.submit parsl.executors.status_handling INFO: Allocated block ID 6594 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6594 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6594 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6594 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6594 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6594.1709794037.9517171.submit parsl.executors.status_handling INFO: Allocated block ID 6595 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6595 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6595 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6595 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6595 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6595.1709794037.9521406.submit parsl.executors.status_handling INFO: Allocated block ID 6596 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6596 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6596 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6596 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6596 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6596.1709794037.9525535.submit parsl.executors.status_handling INFO: Allocated block ID 6597 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6597 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6597 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6597 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6597 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6597.1709794037.952979.submit parsl.executors.status_handling INFO: Allocated block ID 6598 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6598 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6598 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6598 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6598 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6598.1709794037.9533997.submit parsl.executors.status_handling INFO: Allocated block ID 6599 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6599 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6599 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6599 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6599 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6599.1709794037.9538143.submit parsl.executors.status_handling INFO: Allocated block ID 6600 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6600 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6600 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6600 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6600 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6600.1709794037.9542356.submit parsl.executors.status_handling INFO: Allocated block ID 6601 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6601 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6601 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6601 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6601 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6601.1709794037.95465.submit parsl.executors.status_handling INFO: Allocated block ID 6602 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6602 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6602 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6602 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6602 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6602.1709794037.9550688.submit parsl.executors.status_handling INFO: Allocated block ID 6603 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6603 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6603 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6603 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6603 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6603.1709794037.955483.submit parsl.executors.status_handling INFO: Allocated block ID 6604 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6604 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6604 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6604 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6604 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6604.1709794037.9559064.submit parsl.executors.status_handling INFO: Allocated block ID 6605 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6605 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6605 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6605 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6605 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6605.1709794037.9563289.submit parsl.executors.status_handling INFO: Allocated block ID 6606 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6606 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6606 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6606 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6606 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6606.1709794037.9567447.submit parsl.executors.status_handling INFO: Allocated block ID 6607 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6607 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6607 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6607 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6607 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6607.1709794037.9571702.submit parsl.executors.status_handling INFO: Allocated block ID 6608 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6608 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6608 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6608 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6608 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6608.1709794037.9575803.submit parsl.executors.status_handling INFO: Allocated block ID 6609 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6609 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6609 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6609 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6609 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6609.1709794037.9579973.submit parsl.executors.status_handling INFO: Allocated block ID 6610 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6610 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6610 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6610 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6610 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6610.1709794037.958416.submit parsl.executors.status_handling INFO: Allocated block ID 6611 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6611 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6611 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6611 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6611 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6611.1709794037.9793928.submit parsl.executors.status_handling INFO: Allocated block ID 6612 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6612 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6612 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6612 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6612 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6612.1709794037.9799483.submit parsl.executors.status_handling INFO: Allocated block ID 6613 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6613 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6613 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6613 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6613 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6613.1709794037.980379.submit parsl.executors.status_handling INFO: Allocated block ID 6614 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6614 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6614 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6614 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6614 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6614.1709794037.9807959.submit parsl.executors.status_handling INFO: Allocated block ID 6615 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6615 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6615 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6615 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6615 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6615.1709794037.9812224.submit parsl.executors.status_handling INFO: Allocated block ID 6616 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6616 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6616 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6616 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6616 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6616.1709794037.9816425.submit parsl.executors.status_handling INFO: Allocated block ID 6617 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6617 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6617 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6617 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6617 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6617.1709794037.982073.submit parsl.executors.status_handling INFO: Allocated block ID 6618 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6618 2024-03-06 22:47:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6618 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6618 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6618 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6618.1709794037.9824932.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944 parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 734da8249b3695b0cbf3d74de0ee4840 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cb17353f-782b-4df0-beaa-6bf60f5acc4d_biasIsr_1123830_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123830/cb17353f-782b-4df0-beaa-6bf60f5acc4d_biasIsr_1123830_8.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd6133455d1e55d836e7aedf308ad47 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5af87610-064a-46b0-96d3-f24e1092f8cb_biasIsr_1123728_8.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/logs/biasIsr/1123728/5af87610-064a-46b0-96d3-f24e1092f8cb_biasIsr_1123728_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1329 parsl.dataflow.strategy DEBUG: Executor multi has 1329 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 183 more blocks parsl.executors.status_handling INFO: Scaling out by 183 blocks parsl.executors.status_handling INFO: Allocated block ID 6619 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6619 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6619 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6619 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6619 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6619.1709794042.6999812.submit parsl.executors.status_handling INFO: Allocated block ID 6620 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6620 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6620 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6620 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6620 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6620.1709794042.7004652.submit parsl.executors.status_handling INFO: Allocated block ID 6621 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6621 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6621 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6621 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6621 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6621.1709794042.7009053.submit parsl.executors.status_handling INFO: Allocated block ID 6622 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6622 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6622 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6622 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6622 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6622.1709794042.7013357.submit parsl.executors.status_handling INFO: Allocated block ID 6623 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6623 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6623 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6623 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6623 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6623.1709794042.7017589.submit parsl.executors.status_handling INFO: Allocated block ID 6624 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6624 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6624 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6624 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6624 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6624.1709794042.7021916.submit parsl.executors.status_handling INFO: Allocated block ID 6625 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6625 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6625 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6625 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6625 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6625.1709794042.7026205.submit parsl.executors.status_handling INFO: Allocated block ID 6626 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6626 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6626 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6626 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6626 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6626.1709794042.7030559.submit parsl.executors.status_handling INFO: Allocated block ID 6627 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6627 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6627 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6627 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6627 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6627.1709794042.7034883.submit parsl.executors.status_handling INFO: Allocated block ID 6628 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6628 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6628 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6628 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6628 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6628.1709794042.7039108.submit parsl.executors.status_handling INFO: Allocated block ID 6629 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6629 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6629 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6629 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6629 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6629.1709794042.7043357.submit parsl.executors.status_handling INFO: Allocated block ID 6630 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6630 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6630 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6630 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6630 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6630.1709794042.7047563.submit parsl.executors.status_handling INFO: Allocated block ID 6631 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6631 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6631 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6631 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6631 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6631.1709794042.705186.submit parsl.executors.status_handling INFO: Allocated block ID 6632 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6632 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6632 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6632 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6632 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6632.1709794042.7056122.submit parsl.executors.status_handling INFO: Allocated block ID 6633 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6633 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6633 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6633 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6633 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6633.1709794042.70604.submit parsl.executors.status_handling INFO: Allocated block ID 6634 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6634 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6634 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6634 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6634 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6634.1709794042.706462.submit parsl.executors.status_handling INFO: Allocated block ID 6635 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6635 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6635 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6635 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6635 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6635.1709794042.7068877.submit parsl.executors.status_handling INFO: Allocated block ID 6636 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6636 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6636 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6636 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6636 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6636.1709794042.7073176.submit parsl.executors.status_handling INFO: Allocated block ID 6637 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6637 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6637 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6637 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6637 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6637.1709794042.7077413.submit parsl.executors.status_handling INFO: Allocated block ID 6638 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6638 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6638 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6638 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6638 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6638.1709794042.7081778.submit parsl.executors.status_handling INFO: Allocated block ID 6639 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6639 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6639 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6639 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6639 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6639.1709794042.708597.submit parsl.executors.status_handling INFO: Allocated block ID 6640 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6640 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6640 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6640 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6640 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6640.1709794042.7090228.submit parsl.executors.status_handling INFO: Allocated block ID 6641 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6641 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6641 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6641 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6641 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6641.1709794042.7094438.submit parsl.executors.status_handling INFO: Allocated block ID 6642 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6642 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6642 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6642 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6642 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6642.1709794042.7098715.submit parsl.executors.status_handling INFO: Allocated block ID 6643 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6643 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6643 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6643 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6643 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6643.1709794042.7102985.submit parsl.executors.status_handling INFO: Allocated block ID 6644 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6644 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6644 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6644 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6644 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6644.1709794042.7107234.submit parsl.executors.status_handling INFO: Allocated block ID 6645 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6645 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6645 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6645 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6645 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6645.1709794042.7111504.submit parsl.executors.status_handling INFO: Allocated block ID 6646 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6646 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6646 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6646 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6646 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6646.1709794042.7115722.submit parsl.executors.status_handling INFO: Allocated block ID 6647 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6647 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6647 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6647 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6647 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6647.1709794042.7250962.submit parsl.executors.status_handling INFO: Allocated block ID 6648 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6648 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6648 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6648 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6648 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6648.1709794042.7255745.submit parsl.executors.status_handling INFO: Allocated block ID 6649 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6649 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6649 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6649 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6649 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6649.1709794042.7260168.submit parsl.executors.status_handling INFO: Allocated block ID 6650 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6650 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6650 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6650 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6650 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6650.1709794042.7264485.submit parsl.executors.status_handling INFO: Allocated block ID 6651 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6651 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6651 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6651 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6651 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6651.1709794042.7268782.submit parsl.executors.status_handling INFO: Allocated block ID 6652 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6652 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6652 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6652 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6652 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6652.1709794042.7273054.submit parsl.executors.status_handling INFO: Allocated block ID 6653 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6653 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6653 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6653 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6653 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6653.1709794042.7277274.submit parsl.executors.status_handling INFO: Allocated block ID 6654 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6654 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6654 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6654 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6654 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6654.1709794042.7281609.submit parsl.executors.status_handling INFO: Allocated block ID 6655 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6655 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6655 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6655 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6655 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6655.1709794042.728582.submit parsl.executors.status_handling INFO: Allocated block ID 6656 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6656 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6656 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6656 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6656 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6656.1709794042.7290134.submit parsl.executors.status_handling INFO: Allocated block ID 6657 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6657 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6657 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6657 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6657 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6657.1709794042.7294374.submit parsl.executors.status_handling INFO: Allocated block ID 6658 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6658 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6658 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6658 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6658 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6658.1709794042.7298641.submit parsl.executors.status_handling INFO: Allocated block ID 6659 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6659 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6659 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6659 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6659 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6659.1709794042.730288.submit parsl.executors.status_handling INFO: Allocated block ID 6660 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6660 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6660 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6660 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6660 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6660.1709794042.7307158.submit parsl.executors.status_handling INFO: Allocated block ID 6661 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6661 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6661 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6661 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6661 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6661.1709794042.7311478.submit parsl.executors.status_handling INFO: Allocated block ID 6662 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6662 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6662 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6662 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6662 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6662.1709794042.731572.submit parsl.executors.status_handling INFO: Allocated block ID 6663 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6663 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6663 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6663 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6663 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6663.1709794042.7320108.submit parsl.executors.status_handling INFO: Allocated block ID 6664 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6664 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6664 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6664 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6664 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6664.1709794042.7324317.submit parsl.executors.status_handling INFO: Allocated block ID 6665 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6665 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6665 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6665 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6665 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6665.1709794042.7328598.submit parsl.executors.status_handling INFO: Allocated block ID 6666 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6666 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6666 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6666 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6666 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6666.1709794042.733283.submit parsl.executors.status_handling INFO: Allocated block ID 6667 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6667 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6667 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6667 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6667 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6667.1709794042.7337022.submit parsl.executors.status_handling INFO: Allocated block ID 6668 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6668 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6668 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6668 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6668 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6668.1709794042.7341275.submit parsl.executors.status_handling INFO: Allocated block ID 6669 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6669 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6669 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6669 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6669 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6669.1709794042.7345462.submit parsl.executors.status_handling INFO: Allocated block ID 6670 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6670 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6670 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6670 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6670 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6670.1709794042.7349677.submit parsl.executors.status_handling INFO: Allocated block ID 6671 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6671 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6671 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6671 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6671 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6671.1709794042.7353861.submit parsl.executors.status_handling INFO: Allocated block ID 6672 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6672 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6672 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6672 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6672 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6672.1709794042.7358217.submit parsl.executors.status_handling INFO: Allocated block ID 6673 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6673 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6673 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6673 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6673 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6673.1709794042.7362528.submit parsl.executors.status_handling INFO: Allocated block ID 6674 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6674 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6674 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6674 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6674 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6674.1709794042.73668.submit parsl.executors.status_handling INFO: Allocated block ID 6675 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6675 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6675 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6675 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6675 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6675.1709794042.7371156.submit parsl.executors.status_handling INFO: Allocated block ID 6676 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6676 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6676 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6676 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6676 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6676.1709794042.7375355.submit parsl.executors.status_handling INFO: Allocated block ID 6677 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6677 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6677 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6677 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6677 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6677.1709794042.7510622.submit parsl.executors.status_handling INFO: Allocated block ID 6678 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6678 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6678 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6678 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6678 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6678.1709794042.7515404.submit parsl.executors.status_handling INFO: Allocated block ID 6679 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6679 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6679 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6679 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6679 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6679.1709794042.7519794.submit parsl.executors.status_handling INFO: Allocated block ID 6680 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6680 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6680 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6680 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6680 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6680.1709794042.7524097.submit parsl.executors.status_handling INFO: Allocated block ID 6681 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6681 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6681 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6681 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6681 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6681.1709794042.7528353.submit parsl.executors.status_handling INFO: Allocated block ID 6682 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6682 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6682 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6682 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6682 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6682.1709794042.7532585.submit parsl.executors.status_handling INFO: Allocated block ID 6683 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6683 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6683 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6683 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6683 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6683.1709794042.7536757.submit parsl.executors.status_handling INFO: Allocated block ID 6684 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6684 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6684 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6684 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6684 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6684.1709794042.7541153.submit parsl.executors.status_handling INFO: Allocated block ID 6685 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6685 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6685 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6685 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6685 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6685.1709794042.7545412.submit parsl.executors.status_handling INFO: Allocated block ID 6686 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6686 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6686 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6686 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6686 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6686.1709794042.7549725.submit parsl.executors.status_handling INFO: Allocated block ID 6687 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6687 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6687 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6687 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6687 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6687.1709794042.7553997.submit parsl.executors.status_handling INFO: Allocated block ID 6688 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6688 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6688 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6688 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6688 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6688.1709794042.7558239.submit parsl.executors.status_handling INFO: Allocated block ID 6689 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6689 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6689 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6689 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6689 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6689.1709794042.7562416.submit parsl.executors.status_handling INFO: Allocated block ID 6690 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6690 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6690 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6690 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6690 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6690.1709794042.7566602.submit parsl.executors.status_handling INFO: Allocated block ID 6691 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6691 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6691 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6691 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6691 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6691.1709794042.757084.submit parsl.executors.status_handling INFO: Allocated block ID 6692 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6692 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6692 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6692 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6692 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6692.1709794042.7575052.submit parsl.executors.status_handling INFO: Allocated block ID 6693 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6693 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6693 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6693 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6693 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6693.1709794042.7579343.submit parsl.executors.status_handling INFO: Allocated block ID 6694 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6694 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6694 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6694 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6694 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6694.1709794042.7583535.submit parsl.executors.status_handling INFO: Allocated block ID 6695 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6695 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6695 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6695 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6695 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6695.1709794042.758771.submit parsl.executors.status_handling INFO: Allocated block ID 6696 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6696 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6696 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6696 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6696 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6696.1709794042.7592034.submit parsl.executors.status_handling INFO: Allocated block ID 6697 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6697 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6697 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6697 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6697 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6697.1709794042.7596273.submit parsl.executors.status_handling INFO: Allocated block ID 6698 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6698 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6698 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6698 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6698 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6698.1709794042.760053.submit parsl.executors.status_handling INFO: Allocated block ID 6699 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6699 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6699 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6699 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6699 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6699.1709794042.7604735.submit parsl.executors.status_handling INFO: Allocated block ID 6700 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6700 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6700 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6700 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6700 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6700.1709794042.7608955.submit parsl.executors.status_handling INFO: Allocated block ID 6701 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6701 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6701 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6701 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6701 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6701.1709794042.7613144.submit parsl.executors.status_handling INFO: Allocated block ID 6702 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6702 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6702 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6702 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6702 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6702.1709794042.7617285.submit parsl.executors.status_handling INFO: Allocated block ID 6703 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6703 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6703 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6703 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6703 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6703.1709794042.762158.submit parsl.executors.status_handling INFO: Allocated block ID 6704 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6704 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6704 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6704 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6704 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6704.1709794042.7625804.submit parsl.executors.status_handling INFO: Allocated block ID 6705 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6705 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6705 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6705 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6705 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6705.1709794042.7630088.submit parsl.executors.status_handling INFO: Allocated block ID 6706 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6706 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6706 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6706 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6706 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6706.1709794042.763436.submit parsl.executors.status_handling INFO: Allocated block ID 6707 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6707 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6707 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6707 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6707 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6707.1709794042.7769608.submit parsl.executors.status_handling INFO: Allocated block ID 6708 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6708 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6708 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6708 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6708 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6708.1709794042.7774622.submit parsl.executors.status_handling INFO: Allocated block ID 6709 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6709 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6709 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6709 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6709 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6709.1709794042.7779014.submit parsl.executors.status_handling INFO: Allocated block ID 6710 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6710 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6710 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6710 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6710 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6710.1709794042.7783298.submit parsl.executors.status_handling INFO: Allocated block ID 6711 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6711 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6711 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6711 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6711 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6711.1709794042.778752.submit parsl.executors.status_handling INFO: Allocated block ID 6712 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6712 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6712 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6712 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6712 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6712.1709794042.77918.submit parsl.executors.status_handling INFO: Allocated block ID 6713 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6713 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6713 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6713 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6713 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6713.1709794042.7795992.submit parsl.executors.status_handling INFO: Allocated block ID 6714 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6714 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6714 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6714 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6714 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6714.1709794042.780027.submit parsl.executors.status_handling INFO: Allocated block ID 6715 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6715 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6715 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6715 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6715 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6715.1709794042.7804503.submit parsl.executors.status_handling INFO: Allocated block ID 6716 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6716 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6716 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6716 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6716 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6716.1709794042.7808776.submit parsl.executors.status_handling INFO: Allocated block ID 6717 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6717 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6717 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6717 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6717 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6717.1709794042.781305.submit parsl.executors.status_handling INFO: Allocated block ID 6718 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6718 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6718 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6718 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6718 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6718.1709794042.7817245.submit parsl.executors.status_handling INFO: Allocated block ID 6719 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6719 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6719 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6719 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6719 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6719.1709794042.782154.submit parsl.executors.status_handling INFO: Allocated block ID 6720 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6720 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6720 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6720 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6720 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6720.1709794042.7825813.submit parsl.executors.status_handling INFO: Allocated block ID 6721 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6721 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6721 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6721 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6721 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6721.1709794042.783008.submit parsl.executors.status_handling INFO: Allocated block ID 6722 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6722 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6722 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6722 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6722 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6722.1709794042.7834334.submit parsl.executors.status_handling INFO: Allocated block ID 6723 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6723 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6723 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6723 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6723 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6723.1709794042.7838588.submit parsl.executors.status_handling INFO: Allocated block ID 6724 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6724 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6724 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6724 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6724 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6724.1709794042.7842774.submit parsl.executors.status_handling INFO: Allocated block ID 6725 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6725 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6725 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6725 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6725 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6725.1709794042.784694.submit parsl.executors.status_handling INFO: Allocated block ID 6726 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6726 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6726 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6726 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6726 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6726.1709794042.7851188.submit parsl.executors.status_handling INFO: Allocated block ID 6727 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6727 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6727 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6727 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6727 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6727.1709794042.7855394.submit parsl.executors.status_handling INFO: Allocated block ID 6728 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6728 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6728 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6728 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6728 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6728.1709794042.7859638.submit parsl.executors.status_handling INFO: Allocated block ID 6729 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6729 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6729 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6729 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6729 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6729.1709794042.7863812.submit parsl.executors.status_handling INFO: Allocated block ID 6730 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6730 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6730 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6730 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6730 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6730.1709794042.786798.submit parsl.executors.status_handling INFO: Allocated block ID 6731 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6731 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6731 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6731 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6731 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6731.1709794042.7872236.submit parsl.executors.status_handling INFO: Allocated block ID 6732 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6732 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6732 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6732 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6732 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6732.1709794042.7876532.submit parsl.executors.status_handling INFO: Allocated block ID 6733 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6733 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6733 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6733 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6733 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6733.1709794042.7880805.submit parsl.executors.status_handling INFO: Allocated block ID 6734 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6734 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6734 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6734 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6734 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6734.1709794042.7884967.submit parsl.executors.status_handling INFO: Allocated block ID 6735 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6735 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6735 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6735 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6735 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6735.1709794042.7889152.submit parsl.executors.status_handling INFO: Allocated block ID 6736 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6736 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6736 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6736 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6736 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6736.1709794042.789334.submit parsl.executors.status_handling INFO: Allocated block ID 6737 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6737 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6737 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6737 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6737 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6737.1709794042.7897468.submit parsl.executors.status_handling INFO: Allocated block ID 6738 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6738 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6738 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6738 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6738 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6738.1709794042.8032935.submit parsl.executors.status_handling INFO: Allocated block ID 6739 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6739 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6739 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6739 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6739 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6739.1709794042.8037252.submit parsl.executors.status_handling INFO: Allocated block ID 6740 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6740 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6740 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6740 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6740 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6740.1709794042.8041613.submit parsl.executors.status_handling INFO: Allocated block ID 6741 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6741 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6741 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6741 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6741 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6741.1709794042.8045812.submit parsl.executors.status_handling INFO: Allocated block ID 6742 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6742 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6742 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6742 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6742 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6742.1709794042.8050056.submit parsl.executors.status_handling INFO: Allocated block ID 6743 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6743 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6743 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6743 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6743 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6743.1709794042.805421.submit parsl.executors.status_handling INFO: Allocated block ID 6744 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6744 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6744 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6744 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6744 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6744.1709794042.8058481.submit parsl.executors.status_handling INFO: Allocated block ID 6745 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6745 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6745 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6745 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6745 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6745.1709794042.8063285.submit parsl.executors.status_handling INFO: Allocated block ID 6746 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6746 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6746 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6746 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6746 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6746.1709794042.8067594.submit parsl.executors.status_handling INFO: Allocated block ID 6747 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6747 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6747 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6747 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6747 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6747.1709794042.8071945.submit parsl.executors.status_handling INFO: Allocated block ID 6748 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6748 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6748 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6748 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6748 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6748.1709794042.8076167.submit parsl.executors.status_handling INFO: Allocated block ID 6749 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6749 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6749 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6749 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6749 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6749.1709794042.808045.submit parsl.executors.status_handling INFO: Allocated block ID 6750 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6750 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6750 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6750 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6750 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6750.1709794042.808465.submit parsl.executors.status_handling INFO: Allocated block ID 6751 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6751 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6751 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6751 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6751 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6751.1709794042.8088925.submit parsl.executors.status_handling INFO: Allocated block ID 6752 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6752 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6752 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6752 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6752 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6752.1709794042.8093138.submit parsl.executors.status_handling INFO: Allocated block ID 6753 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6753 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6753 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6753 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6753 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6753.1709794042.8097312.submit parsl.executors.status_handling INFO: Allocated block ID 6754 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6754 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6754 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6754 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6754 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6754.1709794042.8101525.submit parsl.executors.status_handling INFO: Allocated block ID 6755 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6755 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6755 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6755 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6755 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6755.1709794042.8105695.submit parsl.executors.status_handling INFO: Allocated block ID 6756 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6756 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6756 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6756 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6756 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6756.1709794042.810998.submit parsl.executors.status_handling INFO: Allocated block ID 6757 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6757 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6757 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6757 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6757 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6757.1709794042.81142.submit parsl.executors.status_handling INFO: Allocated block ID 6758 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6758 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6758 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6758 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6758 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6758.1709794042.811848.submit parsl.executors.status_handling INFO: Allocated block ID 6759 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6759 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6759 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6759 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6759 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6759.1709794042.8122656.submit parsl.executors.status_handling INFO: Allocated block ID 6760 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6760 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6760 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6760 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6760 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6760.1709794042.8126814.submit parsl.executors.status_handling INFO: Allocated block ID 6761 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6761 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6761 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6761 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6761 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6761.1709794042.8131034.submit parsl.executors.status_handling INFO: Allocated block ID 6762 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6762 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6762 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6762 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6762 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6762.1709794042.8135207.submit parsl.executors.status_handling INFO: Allocated block ID 6763 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6763 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6763 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6763 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6763 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6763.1709794042.813953.submit parsl.executors.status_handling INFO: Allocated block ID 6764 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6764 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6764 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6764 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6764 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6764.1709794042.8143744.submit parsl.executors.status_handling INFO: Allocated block ID 6765 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6765 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6765 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6765 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6765 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6765.1709794042.8147929.submit parsl.executors.status_handling INFO: Allocated block ID 6766 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6766 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6766 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6766 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6766 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6766.1709794042.8152175.submit parsl.executors.status_handling INFO: Allocated block ID 6767 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6767 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6767 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6767 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6767 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6767.1709794042.8156333.submit parsl.executors.status_handling INFO: Allocated block ID 6768 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6768 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6768 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6768 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6768 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6768.1709794042.8160648.submit parsl.executors.status_handling INFO: Allocated block ID 6769 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6769 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6769 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6769 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6769 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6769.1709794042.8164911.submit parsl.executors.status_handling INFO: Allocated block ID 6770 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6770 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6770 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6770 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6770 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6770.1709794042.8169239.submit parsl.executors.status_handling INFO: Allocated block ID 6771 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6771 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6771 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6771 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6771 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6771.1709794042.8173442.submit parsl.executors.status_handling INFO: Allocated block ID 6772 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6772 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6772 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6772 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6772 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6772.1709794042.8177576.submit parsl.executors.status_handling INFO: Allocated block ID 6773 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6773 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6773 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6773 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6773 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6773.1709794042.8181787.submit parsl.executors.status_handling INFO: Allocated block ID 6774 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6774 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6774 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6774 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6774 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6774.1709794042.8185923.submit parsl.executors.status_handling INFO: Allocated block ID 6775 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6775 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6775 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6775 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6775 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6775.1709794042.8190105.submit parsl.executors.status_handling INFO: Allocated block ID 6776 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6776 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6776 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6776 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6776 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6776.1709794042.81943.submit parsl.executors.status_handling INFO: Allocated block ID 6777 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6777 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6777 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6777 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6777 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6777.1709794042.819859.submit parsl.executors.status_handling INFO: Allocated block ID 6778 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6778 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6778 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6778 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6778 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6778.1709794042.820276.submit parsl.executors.status_handling INFO: Allocated block ID 6779 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6779 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6779 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6779 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6779 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6779.1709794042.8206913.submit parsl.executors.status_handling INFO: Allocated block ID 6780 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6780 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6780 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6780 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6780 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6780.1709794042.8211205.submit parsl.executors.status_handling INFO: Allocated block ID 6781 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6781 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6781 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6781 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6781 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6781.1709794042.8215413.submit parsl.executors.status_handling INFO: Allocated block ID 6782 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6782 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6782 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6782 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6782 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6782.1709794042.8219721.submit parsl.executors.status_handling INFO: Allocated block ID 6783 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6783 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6783 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6783 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6783 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6783.1709794042.82239.submit parsl.executors.status_handling INFO: Allocated block ID 6784 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6784 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6784 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6784 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6784 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6784.1709794042.8228137.submit parsl.executors.status_handling INFO: Allocated block ID 6785 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6785 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6785 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6785 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6785 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6785.1709794042.8232305.submit parsl.executors.status_handling INFO: Allocated block ID 6786 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6786 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6786 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6786 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6786 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6786.1709794042.823646.submit parsl.executors.status_handling INFO: Allocated block ID 6787 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6787 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6787 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6787 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6787 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6787.1709794042.824069.submit parsl.executors.status_handling INFO: Allocated block ID 6788 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6788 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6788 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6788 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6788 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6788.1709794042.8244853.submit parsl.executors.status_handling INFO: Allocated block ID 6789 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6789 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6789 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6789 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6789 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6789.1709794042.8249073.submit parsl.executors.status_handling INFO: Allocated block ID 6790 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6790 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6790 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6790 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6790 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6790.1709794042.8253233.submit parsl.executors.status_handling INFO: Allocated block ID 6791 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6791 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6791 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6791 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6791 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6791.1709794042.825751.submit parsl.executors.status_handling INFO: Allocated block ID 6792 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6792 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6792 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6792 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6792 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6792.1709794042.8261907.submit parsl.executors.status_handling INFO: Allocated block ID 6793 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6793 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6793 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6793 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6793 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6793.1709794042.8266132.submit parsl.executors.status_handling INFO: Allocated block ID 6794 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6794 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6794 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6794 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6794 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6794.1709794042.8270395.submit parsl.executors.status_handling INFO: Allocated block ID 6795 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6795 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6795 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6795 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6795 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6795.1709794042.8274589.submit parsl.executors.status_handling INFO: Allocated block ID 6796 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6796 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6796 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6796 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6796 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6796.1709794042.8278835.submit parsl.executors.status_handling INFO: Allocated block ID 6797 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6797 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6797 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6797 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6797 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6797.1709794042.8283005.submit parsl.executors.status_handling INFO: Allocated block ID 6798 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6798 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6798 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6798 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6798 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6798.1709794042.8287158.submit parsl.executors.status_handling INFO: Allocated block ID 6799 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6799 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6799 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6799 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6799 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6799.1709794042.829137.submit parsl.executors.status_handling INFO: Allocated block ID 6800 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6800 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6800 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6800 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6800 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6800.1709794042.8295543.submit parsl.executors.status_handling INFO: Allocated block ID 6801 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6801 2024-03-06 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6801 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6801 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-06 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-06 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-06 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-06 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.246,198.48.92.26,10.64.129.246,n3246 -p 0 -c 1.0 -m None --poll 10 --task_port=54538 --result_port=54537 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/runinfo/000/multi --block_id=6801 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.cluster_provider ERROR: Failed writing to submit script: /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/bias/cpBias/20240306T224255Z/None/parsl.multi.block-6801.1709794042.829975.submit parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23101762020944